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SECTION I 
INTRODUCTION 

Education TurnKey Systems, Inc. and its subcontractor, Kirschner 
Associates, Inc. (KAI) and key University of Maryland and other consultants 
are pleased to submit this report in response to the needs of the National 
Institute of Education as specified in the Request for Proposal (NIE-R-75- 
0022): To develop a Design to Study Individualized Instruction, as part 
of a general study of compensatory education activities. The ultimate 
purpose of this two-phase effort is to provide both Congress and NIE 
with policy- re levant information on the effectiveness and effects of 
well -implemented individualized math and reading programs as they compare 

with similar standardized programs, 
it 

BACKGROUND 

Jhe Educdtion Amendinentb of 1974 provide Por a IhojOuyh studj^of 

compensatory education.* Specifically, Clauses (1) through (5) of Section 

821 (a) of P.L. 93-380 mandate the Natioi?al Institute of Education to 

conduct a study which will include: (1) an examination of the purposes 

and the effectiveness of such programs; (2) an analysis of the means to 

identify the children in need of such programs ; (3) an analysis of the 

effectiveness of methods and procedures; (4) an examination -of alternative 

' methods; (5) the administration! by NIE, of not more than -20 experimental 

programs; and (6) findings and racommendations. 

• The Research, Plan, Compensatory Education „Study (16 December 1974, 
- • ' " « 

pp. 3-4) describes the three basic components of the overall study (of which 

District Survey, II as modified is a part) as: 

* » 

- - — - - - " ■ .1' ' 

* See TUFfNKEY, Proposal , -June 6, 1975, hereafter referred to as Proposal. 

G 



"Research on Programs for students in need of compensatory 
education. ... 

"Research on the allocation of funds to support compensatory 
^education programs... 

. "Research on the operation of federal, state, and local agencies 
\ which deliver compensatory education programs." 

The large-scale NIE study of compensatory education is unique in 

severaV important respects — specificity, tone, ajid context. Congress 

has'' mandated the overall study in unusual specif^icity : a) policy research 

directed to specific questions or issues; b) prior Congressional review 

of the proposed design and plan; c) solicitation of advice from the National 

Advisory Council on Education of Disadvantaged Children in the design 

and operationar"phases; and d) restrictions regarding review of study 

findings prior to submission to Congress. Hende, the mechanics and 

procedures of this study differ significantly from others conducted in this 

field. 

Implicit in these specified characteristics is a unique tone the 
desire for a comprehensive study of compensatory education more relevant to 
Congressional needs. During the hearings and subcommittee sessions on the 
1974 Amendments to ESEA, several different approaches and techniques for com- 
pensatory education received much more interest than in the past, including: 
a)'educational deficiency as me^asured by criterion-referenced tests as the 
basis for allocation of funds; b) contingency or .incentive funding based upon 
•program or student achievement; c) alternative procedures and mechanisms for 
administering/monitoring compensatory education programs; and d) evolution 
of national standards of educational .excellence and equity beyond the concept of 
equal opportunity. 'Policy an(^" cliscipl inary research findings emanating from 
this study should provide useful information for the policy formulation process. 



The unusual tone of this study can be attributed to several reasons, including 
frustrations of key Congressmen due to the lack of a consensus of which pro- 

" grams work best and why» and confusion (and hence, criticism) as to who should 
demonstrate results — local education agency (LEA) program staff or those 
conducting evaluations?' An awesome responsibility is pr'esented to *NrE and its 
contractors: this particular research effort must be as scientifically flaw- 
less as it is feasible in design and execution, so that any lack of evidence 
of success can be attributed to programs or their implementation, not to the 
research effort itself. 

The context of the overall study differs from most others. Supported 
by the results "of recent state education^ agency compensatory education studies 

^ in California, Michigan, and elsewhere, and other .information which suggests 
that compensatory education can work if it i§ individualized, Congress" has 
clearly expressed its intent for the development and implementation of indi- 
vidualized prograriis cf instruction for students. A study cf this, assui^ption 
must, therefore', be carefully designed, comprehensive ia scope, cautious in 
attribution, and insightful in suggesting feasible ways for further'improve- 
ment. 

PURPOSE OF THE STUDY 

In the'RFP, NIE has specified the' overall purpose and characteristics 
of this study: , > ' 

"...to compare for compensatory education programs teac^hing reading 
and mathematics, the effectiveness of well -implemented individua- 
lized instructional programs with -that of well -implemented stand- 
ardized programs. The focus of the study will be on assessing 
the effects of individualization and not on the variety of addi- 
N tional dimensions along which programs may vary. It will employ 



• • • <; • . • . 

in-depth observatioVis of programs to fully describe- their 'operating 
characteristics, to determine the degree of implementation, ana 
to describe settings where adequate implementation may be diffii- 
cult to achieve. PrograrfTs will be, assessed by measures of reading 
and mathematics achievement and by their broader effects on^lass^ 
room environment." 



Individualization of instruction has been identified for specific atten- 
tion because of the expansion of the concept in education generally, enthur 
siastic support of education researchers and practitioners, and interest -. 
shown by Wngress. Clause B of Section 821 (a) of P.L. 93-3^0 mandates NIE^ 
to conduct "an analysis of the effectiveness of methods and procedures for 
meeting educational needs of children, including the- use of . individual writ- 
ten educational plans for children." . The House ConiiTi tte^ ^n E.ducation and ; ; 
Labor Report. .to Accompany Hr'59 (21 February 1974) stated the intent of; . ^ 
Congress regarding individualized ins;truction very clearlit^ -^^^ * 

"The Committee Bill amends' Title I to. includes sense of .Congress . 
statement that local school districts are encouraged to dev,elop 
isftiividually describee! plans, for alTchildren particrpatmg in 
• • ' Title L ''^rogr?a!iis . These plans arp nnt meant -to be reauired 
■ ^' ^ ' under. Title Vbut the adoption of this statement is meant to show 
• '4" the feeling of Congress that there'is a need for more individualized 
educatiofTafid a- need for more systematic planning in education to 
include the involvement. of teachers, parents, and child." 

It should also be noted that in the minority dissenting views, no' member of 
the Coimittee took exception to the "sense of Congress-" as stated_above, even 
though other issues, such' as sex education, textbook censorship, parental 
involvement, and other sections of 'the 1974 Amendment, were- debated; In 
light of this apparent unanimity it is obvious that -Congress has a commit- 
ment to individualized instruttion and t/ants to learn more about its effect- 
iveness and the. conditions under which .it helps disadvantaged students moat. 

Some of the more important policy questions to be addressed which emerge 
upon a review of the RFP and the Hearings and Testimony leading to^he passage 
of thie'1974 Amendments are as follows: 
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• How effective are wel 1- implemented individualized programs in raising 
achievement in math and reading for disadvantaged children? The unit 
of comparison would be well -implemented standardized or traditional 
type compensatory education programs with similar objectives and 
similar compensatory education students. 

• What is the impact and interrelationship among other effects created 
in the classroom and school environment that can be attributed to 
the effective implementation of -individualized instruction? These 
general effects would include'teacher morale and expectations, 
sense of involv,ement, classroom discipline, student-attitudes, and' 
time spent on productive activities, among others. 

• To what extent do local education agencies utilizing well-implemerlted 
indiv4dualized programs of instruction meet the specific needs of • 
individual students? What types of encouragement or what types of 
"barriers need to be removed in order for LEAs to do this job more ' 
effectively?. As the. House Committee Report noted, "The Committee 

^, feels strongly that the local school agency is the appropriate level 
.' • to determine the specific needs of educationally deprived children 
. and should ti^ primarily responsible for determining app^oaches to 
meet those needs" (page 2l) . 

• To what extent does individualized instruction facil itate' or provide " 
an atmosphere conducive to the furtherence of additional Congressional 
intents such as: a) the degree, extent, and types of parental involve- 
ment; b) the targeting of compensatory education funds or. schools with 
high concentrations of eligible students; c) the facilitatioii of 
higher quality evaluations to provide feedback for programs improve- 
ment purposes; and d) the facilitation of more effective and higher 
quality planning and directly related teacher training at the LEA 

and building level? 

Perhaps as important as the issues to be addressed in this study are those 
which are not to be addressed. For example, this study will not attempt to " 
determine the degree to which individualization exists nationally in compen- 
satory education programs nor the type and nature of these programs. Nor is 
this study designed to analyze, and rank the various individualized instructional 
"packages" and curriculae in terms of their relative cost-effectiveness. ' 

Since this study is part of the overall study of compensatory education 
conducted by NIE and is related to District Survey I, a discussion of the 
relationship between the two wi]l assist in focusing upon the specific 
issues to be addressed. 
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RELATIONSHIP TO DISTRICT SURVEY I 

District Survey I is designed to be a general survey of compensatory 
education activities in a sample of districts, examining^ a) the purposes 
of compensatory education programs, as stated and perceived at all levels; 

b) 'the planning and coordination mechanisms used to address these purposes; 

c) operating characteristics and procedures followed in meeting objectives ^ 
and d) evaluation activities and techniques used. The major questions to, 
be researched^ analyzed, and answered in District. Survey I are: 

1. ^What are the purposes of existing compensatory education (comp 

ed) programs, as stated and perceived, -by education agencies,' 
staff, and parervts? , > - 

2. What types of plannifvg procedures, and coordinating mechanisms 

are used to ensure efficient use of funds and program in- * ^ 
tggration into regular school activities? To what extent have' 
LEAs developed programs .'cordis tent with and relevant zo stated 
goals and objectives? Are object^ives realistic and programs i 
feasible? ' . ' 

3. What are the levels of funding and the nature of the allocation' 
of these funcis? What are the operating CMuruCtsnstics Oi the. 
various types of comp ed programs as/they reflect instructional 

* coordination mechanisms, training, naturae and extent'of staff 
participation, and associated problems? 

' 4\ What is the'nature and extent of evaluation^ activities pre- 
scribed and used to assess^ prograTii success,' including type " 
(i.e,, summative or process), criteria and measure, design* 
. (e.g., pre/post, post-test only), type of evaluation* (e-.g. , 

internal, third party),' feedback (e.g., extent and use), ^ 
and reporting (e.g., util ity, timeliness , relevancy)?, How 
successful have various 'types of programs been in making 
modifications in response to LEA anrf SEA evaluation results? 

The purpose of this study is to assess the effectiveness of individualized 
instruction as it is currently used in compensatory programs tn schools. 
The study is not meant to be a comprehensive survey like District Survey 
I, but rather a concentrated examination of selected programs which provide 



iRidvidualized classroom instructictn; The ntejor foetus will be on assessing, 
progr.am success in the areas of reading and mathematics achievement, and 
general implications of individualized instruction^ for "^he classroom 
environment, while District Survey- I has a much broader scope. 
: However, as explained later, NIE has requested that the design of 
. this study be.coordirTated with that of District Survey I regarding: ^ 
a*) coordinated site^elgction; and b) "validation" of data collection 

• instruments. Fn order. to ensure effective coordination, as described 
in 'our Prbp&sal, dif feVences in definitions of compensatory education 

'"programs" and "student" (p. I-8)> "educational plans" tp. 1-7), and 

* "individualized plans of instruction" (p. 1-9) will have to.be resolved. 
Moreover, the overcill time schedules of the two studies must be, compact iWe. 
These considerations are discussed in greater detail in the task descrip- 
tions -in Section 3, ^ * 

SUMMARY OF RE^SEARCH ON INDIVIDUALIZED INSTRUCTION • 
A. Status of Individualization of Instruction 

« In order to assist in refining the overall design a rather e-xtensive 
review wa-s conducted to determine the status- of research on individualized, 
instruction. While this survey and research was not as j*n1:ensive as was 
desirable due to the relatively short timeline,, more than 75 reports ancj : ^ 
documents were, reviewed and discussion? were conducted with over 50 in- 
dividuals who have been intimately involved^^ tiie_Jteve1opmen 
tion ofindividualized instructional programs ''across the country! Below, 
we summarize the results and findings o'^ this survey, identifying the 
^ implication^ for the 'oyerall st^dy design and the. design approach 'we are 
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pr*opos>ng in the subsequent chapter/-^ those instances where .the 
findings noted below were^stated in greater detail in our original 
l^roposaJ , we have so indicated with appropriate references? 

I. While the philosophical differences between individualized ^ 
and standardized instruction are generally easy to describe, the 
philosophical differences among individualized programs in math 
and reading a^e significant . - ^\ 

Traditional or standardizefd instruction perceives the 

. ' , /■ 

class, or group as an entity itself. .Each student is presumed to' 

have relatively equal learning needs, abilities, and responses. 

Instruction is teacher-paced and scheduled to meet' the convenience 

of the school and the teacher. Students taught by traditional or 

standardized techniques are generally given the same assignment 

' regardless of individual capabilities or progress. Where individual 

assignments are made they are usually to be completed in a specific 

time period. ^ .^-^^-^ ' 

On the other, hand, the philosophy urid.erlying individualized 

; instruction recognizes that each individual Vearns his/her own way, 

and his/her own pace. Individualized instruction therefore seeks 

to motivate the child by finding areas of interest and prescribing 

individual assignments based upon the child's interest areas. As 

a student-oriented approach, it requires a diagnosis of individual 

students to determine the kind of learning experiences needed and 

then to prescribe^ programs that are mostly pupil-directed and pupil- 

admtnistered within general limitations. 

, 1-8 • ' ^ - 
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In reviewing the various instructional systems and critiques 
of many of the same systems conducted by others (e.g.', DeVault and 
EPIE), the philosophical foundations underlying the various in- 
. Jividualized learning systems Within the context of the distinction 
made above do vary, if npt in kind then certainly in degree. 

First, the locus of control and decision-making underlying 
several individualized learning systems varies considerably." Several 
Follow Through models (e.g.. Far West Regit)na1 Education Laboratory), 
several contingency-based frtfJdels (e,g.. Grand Rapids Contract Learning 
Project), and, to some extent, "packaged'* systems (e.g.. High . h 
Intensity Learning Systefn), allow for greater student involvement in: 

a) selecting instructional objectives and/or activities including; 

b) selecting reinforcement activities; c) accepting responsibility 
for pacing; and d) jecording progress. Other systems (e.g., the 
Uiii ver^sity oT Oregofi FgIIow Through rnodel , New Century) , which re-, 
quire ijitensive pre-service training, are designed to insure greater 

• control by the teacher in activities such as: a) prescribing 

specific activities and enrichment materials; b) instructional 

management, including record keeping; and c) applying motivational 

* 

techniques » 

Second, approaches to individual ing classroom i nstruction 
also vary. The IGE process represents an illustration of the. "top- 
down" approach for individualizing math and reading programs through 
an extensive^^prepa^^ includes participation in a 

national "network" or "league" creating change in the district 
and school environment. The specific individualized instruction 

_ 1-9 
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program, or the one which is developed by the staff, is of lesser 

importance than creating an environment conducive to individualiza- 
tion. On the other hand, learning systems such as BRL and New 
Century utilize a pilot program approach in individual buildings 
with the focus on individualizing instruction in specific classrooms 
("b"ottom-up" approach) and over time "changing other environmental 
conditions and factors which can facilitate a more effective implemen- 
tati6n of the specific program. While over time many of the same 
probj^ems are confronted and eventually solved, the immediacy of 
resolution varies- 

r 

2. Definitions of individualized instructional programs 
vary in terms of scope and specificity; existing classification 
schemes also vary accordin to intended purpose . 

During early review of the literature on individualized 
instruction and subsequent discussions with LEA staff and technical 
consultants, it became apparent that no uniform definition of 
individualized instruction is accepted. Moreover, any definition 
to be used in the study would have to be used for the purposes of 
the study, identifying those characteristics which would be the 
focus of the study. The RFP proposed several characteristics of 
individualized programs, including the existence of stated performance 
objectives, speci^al procedures for diagnosis and prescription, 
the use of progress tests, and, most critically, student, progress 
in remedial instruction which is based on student performance on 
progress tests and allows students to proceed at their own rate. 
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^ ' ■ . 'One of the first attempts to develop a set of procedures 
and classification scheme was developed by DeVault, et.al. (1973); 
a chart describing this, descriptor is included in Exhibit I-l . 
After field testing, he indicated that.the instrument was useful 
in describing programs as they exih and the degree to-which in- 
dividualizati9n in a program' could be described in accordance 
with standards established b^ the^ individual teacher. The 'instru- 
ment is limited, however, in comparing various programs in terms ^ 
of individualization. Since the early 1970's, EPIE has developed 
two instruments-, one a hybrid version of the other to classify 
programs along 15 items , -providing a range of variables (EPIE, 1972," 
see'Exhibit 1-2). Subsequently, a variation of this classification 
model was developed and applied to over 25 learning systems in 
accordance with the chaV-acteri sties described in the implementa- 
tion guides (1975). The former instrument was designed to be 
used by practitioners in assessing the degree to which various types ^ 
of materials could be used for individualized instruction, while 
the primary purpose of the latter was- to describe the character^ 
istics of over 25 learning systems. 

A review of these Classifications/descriptors,, as well as 

1 \ 

others (e.g. ,/I/D/E/A/ 's IGE Implementation kit), indicates that 
while it is possible to develop specific descriptors and procedures 
for classifying program characteristics, such descriptors must 
focus upon the specific purpose in mind: hence, while the above 
documents are extremely useful ,• a classification scheme specifically 
designed for the purposes of this study was needed. This. scheme 
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Exhibit 1-2 



Classification and Range of VariaTjIe.S'for" 
Assessing Individualizeti Instructiojial Materials 



Constructs and Variables 



Psoduct Characteristics 



A, OBJECTIVES 

1. Sele^ction of Objectives 

2. Specification of Outcome Goals 

3. Populajion Specific 

b/ organization 

4. Structurjg of Subje^ct Matter 

5. Sequence ^ 

6. Scope , « ' 
, " 6.1 Coverage 

6.2 Range ' 

7. Branciiing 

8. Recy€linff ^ 

9. Selection of »Vteterials 

9.1 Supplementary Materials 
9.^ Selection of Materials 

10. Learning Envlropment 

C. METHODOLOGY . 

I'O. Learning Environment 

11, Met-hodology 
. 12. Time 

13. Pacing 

14. Modes " 

' *** 

D. EVALUATION 

15/ Evaluation of Learning 
' 16. Evaluation Approaches 



Explicit 
Fixed 
Fixed 



Defined 
Flyed Entry 

Fixed Coverage 
Narrow Range 
Linedf Rouie 
Limited 

Limited ^ 

Explicit 

Explicft 



Closed 
Structured 
Fixed 
Fixed 

Limited 



Norm-Referenced 
Teacher-Centered 



Vague 
Variable . 
Vatiabie 



Implicit,. Not Discernible 
Differentiated Entry 

. .Flexible Coverage 
Broad Range 
#MLrltlple Roule 
Multiple^ 

Many 
Open 
Implicit 



Open 
* Eclectic 
Flexible 
Variable* 
Multiple ' 

^ Criterion-Referenced 
Pupil-Centered 



Absent 



A 



V 

Absent 
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is described in Section Ill/Tasks 1 and 2 of this Report* 

3. Operating characteristics vary considerably among 

individua]\e/ learning systems, either by design or for other reasons . 

While this finding is implicit in our earlier observations,' 

-^*the nature and degree of'variation in critical areas is apparent 

from a review of implementation guides and observations of a number 

of operating programs. And as noted below, these differences have 

major implication? for the study design. The primary implication 

* for this study is the need ,tp "capture"' the--quali t^tive differences 

among programs, especially in terms of implementation variables. 

Five pH^njcipal categories encompass these operating characteristics: 

^a! Diagnosis/Prescription 

One .critical element in the individualization process is that of 
diagnosis/prescription. In practice, procedures vary. "Tests" 
that are administered may be subjetti^ve tests, standardized norm- ^ - 
referenced, or cri teria-refere'nced tests, specifically de^jgne'd' 

lUf 'viiv^ . ^1 v/^i uiii . f\(iUijroi^ uilu 1 11 1 pi u uu u 1 Ui I Cun ouocu upuii oc::)o 

scores or subjective evaluation. Prescription of instructional 
approaches and materials may be based upon a pre^prrog rammed procedure 
' (e.g. ,. University of Kansas Follow Through, Project PLAN) which can 
be computer-based or can be developed more sutjjectively by ^individual 
teachers or groups of teachers- (e^. , Grand Rapids Project Target). 
In most instances some subjective elements based on teacher inter- 
pt'etations .are followed. ^ 

b. Mastery/Proficiency Levels 

In the vast majority of individualized programs, mastery .or 
proficiency levels are prescribed, recommended, and/or developed 
in order to' determine how well students have mastered particular 
materials prior to branching or continuing the prescribed course 
of instruction. Mastery tests or exercises may be either "paper and 
pencil" or "performance-based," covering specific learning objectives 
or skills such as word attack, decoding, etc. Specified criteria 
for mastery may' include a specific number of items for a skill or 
performance objective (e.g., three out of four i.tems). At* the lower 
.grade levels (i.e., K-1), the number of items per objective will 
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usually be fewer than at the higher elementary grade ]evels.- In 
other instances, the mastery level may require satisfactory 
completion of all items associated with that skill or learning 
objective/ In some programs, up to three forms of a particular - /' 
set af items will be developed for each objective, for diagnosis J 
progress checks, and "post-test" reporting. Many factors influence 

.the mastery or profi.ciency levels used. Where minimal perfonnance^ 
ol^jectives are specified by the SEA or district and contingencies 
such as subsequent funding are based upon the "achievement' gf 
student" (e.g., in the Michigan Chapter 3 Compensatory/ModeH.,-: - ' 
the number of items to be successfully completed in dete^inlng 
mastery- will be relatively few. Conversely, when fiji^cllng con-^ 
tingencie^ do not exist and teacher aspirations ar^' higher, such 
as in the initial year of implementing a teacher- d.eVe1oped i4i- ' 
dividualized program, the ^niwiber of objectives_^^be:iiiastff^d ,as 
well as the criterion levels will often'be unr'iaTlsticariy high. 

'During the program selection and analysis phase Sf- this^siudy, 
these variances will be adciressed. ' 7 ' 

c. Degree of Teacher Discretion. ^- - 

The degree to which^ the progranj is designed to be "teacher proof" 
varies among programs.- \x\ certain typaSr oT injfividjuaTized ^ 
(e.g., J^ew Century), the impl ementatton^p^larLfs'^xtremeiy specif ic 
as regards the utilization of materials and Jeacfrer .self-evaluation 
instruments. In others (e.'g., FrojrctfPLAN) , tea Cher- training 
is limited to the implementation gf 4^s--spec3frig^p At 
the ^t her end of the' conti nuum are fyr^gralns fe.g.. Grand Rapids 
Project Target) which merely stjafitor^-t^ejia of^teacher involve 
ment in designing and implementin5?the instructtonal program. 
For e^cample, the "system" mi^t corislrst of an e-Xtensive teacher 
training program .and a taxonomy, of -objectives keyed to a variety 
of source materials (e.g. ^ HltSj^Ihe teacher is t?ra>hed to design 
the program, select and categorize materials, develop and/or modify 
tests, etc. The variances described a'bove raise critical issues • 
in determining the degree to which an individualized instructional 
program is wel 1-implemenled. For example, in the New Century 
program, a baseline for determining deviations from the suggested 
implementation .pla-n would be easy to identify. However, in other 
programs with very general implementation procedures leaviiig much 
discretion to the teacher, there exists very little baseline in 
determining the degree to wFiich the program is actually implemented 
in accordance with a^plan. This issue is addressied in our sample 
design discussion. 

d. Content Sequences 

Evaluators attempting to compare the effectiveness of .various 
individualized programs have often found^that different programs ■ 
emphasize 'the development of different s1<:i1ls at different levels 
with different degrees of intensity. ^ In matheifiatics , the Gatteno 
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^"adaptive approach" is based on individual learning styles ,^ 'directed 
by a highly trained teacher. Over a short period of time it would 
te difficult if not impossible to compare this prograoi with^the BRL - 
math program, for. instance, which is structured and sequenced / 
differently. In'reading, certain systems er];Tphasize the development 
of specific skills at specific grade levels over a very shQft but 
intensive period whereas others develop a number of skills almost 
concurrently over a longer period of time. Since the content 
and sequencing af instruction Vary among programs^ care must be 
taken tb ensure that these coYisiderations' are taken into account in 
the selection of Sample and instruifients, * v, 

e. Classroom Management/Organization 

Classroom management 'and organization of ' individualized programs 
will vary from very structured, inflexible center designs to un- 
structured almost indescribable situations such as those in open 
classrooms or open schools. The results of tbe PREP study conducted ' 
by the USOE in the late sixties found general methods and organizational 
patterns in the 45* schools surv^eyed, including: directed activities 
in multiple or single learning areas under a regular or unscheduled 
time period; and selected activities in multiple or single areas 
with time scheduled or unscheduled. Classroom management techniques 
which have more recently emerged include: computer-managed in- 
structional programs where diagnosis and prescription are conducted 
outside the district with immediate feedback to 'the teacher and 
often the student (e.g., Capital Areas Skill Center reading program, 
Project SCORE); and student-majiaged programs where students keep 
their own records of mastery and the teacher merely spotchecks. 
In most cases the nature of the classroom management/organi zation of 
the individualized programs is d&pendent upon or influenced by: 
the suggested design inherent in the individualized program itself; 
local facilities and building considerations; and the nature of the 
instruction (e.g., high intensity remedial vs. long term develop- 
mental). In many instances the organizational and management 
characteristics reflect the desire to avoid barriers which otherwise 
might impede the effective implementation of individualized instruction. 

4. The majority of individualized "systems" have under- 
gone or are presently undergoing change . 

The information gathered by project staff during the design 
phase of the study indicates. that jnost individualized instructix)nal 
systems have undergone or are presently undergoing significant 
c^ignges in content, structure, and in certain instances, philosophy. 
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'^The nature and reasons for these changes vary among the c^t^ones 
of projects. ' • , 

Publishers appear to be attempting to simplify the 
implementation of the various systems-. "They are also recognizing 
that teachers, where economically feasible, desire a variety of 
educational materials. Hence, several (e.g., HILS) have deveV)ped 
taxonomies of skills keyed to materials other .than the ones that ^ 
they** specifically publish. Moreover, what might have been advertised 
as a complete and total learning system several years ago has now 
^ been broken up into different "products'*'. For example, Westinghouse 
Learning Corporation will now provide the performance objectives , 
the test program (SCORE), as well as the teaching-learning units, 
as individual "products" emanating from Project PLAN. HILS in- 
cludes not only ? taxonomy of objectives and source materials but 
also provides services whereby a district can purchase a computer 
printout of the specific oty^ectives covered by the materials in a 
specific classroom. Publishers such as Prescriptive Learning, Inc., 
Education Projects Corporation/ and Zw^ig Associates (Fountain 
Valley) prwr^e^^^itMJajc^^ products. 

Instructional p.rograms have also become less' programmed . 
in nature, allowing fqr more teacher discretion during implementa- 
tion. The reasons for these trends reflect a combination Of 

increased sophistication on the part of the client (e.g., more 

«i • * 

r 

reading specialists involved in decision-making), general economic 
constraints facing schools, and greater teacher involvement in 
decision-making generally at the building level. 
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A review of materials and discussions with Follow-J"hVough ^ 
Model directors and responsible USOE officials indicat^Jj^at virtually 
all Follow ThroT^gtlPro grams have recently undergone or are presently 
undergoing changes.) Indeed, one of the diff iculities in 
designing the SRI Follow Through evaluation was an attempt to 
determine baseline regarding thd implementation procedures. The 
reasons for these changes include: relevant program feedback, un- 
' expected availability of funds provided to LEAs, and lack of sponsor 
technical assistance and training (e.g., sey^ral Follow Through 
developers have gone out of business). The only exception to the 
rule appears to be the LRDC Follow Through model which according 
to the Project Director, has not undergone any significant change in 
the past two yfears . | 

The most stable group of individualized projects appear to 

tiiu:>c u:>Gu in fcuerciiiy a»iu S ua uc vaiiuaucu iiiuuG'o wiiiv^n nave, 

SeerV^ submitted (or are presently being submitted) to the Dissemination 
Review Panel at USOE. Thjs changes -wbich have occurred in these 
programs have been based upon program feedback and to some extent 
upon general economijc conditions and districtwide policies. Stability 
at' sites nominated by publishers as being well-implemented appears 
to be basically a function of the stability of districtwide policies 
and procedures, particularly in districts which are using many 
different programs (e.g., Dallas, Dade County). In these districts 
.the most^isruptive elements effecting* program stability appear 
to be .implementation of court-ordered- desegregation plans through 
bussing, or transfer of^taff . 

. 23 
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Jhe implications of these factors on sample'selection and 
initial verifica^tion for progfam stability appears to be significant 
indeed, as described in the following Section. 

B, Effects and Effectiveness of Individualized Instruction: Preliminary Findings 

Below, we briefly. review some of the most relevant research on 
individualized instruction, focusing upon: a) the. relationship between 
schpol and classroom environment and individual ize^ instruction, including 
staff, student, and other attitudes; b) the effectiveness of individualized 
programs in math and reading; and c) the relatioi^ship between program 
cost and effectiveness. In our Proposal, we summarized a number of large- 
scale studies and yyesearch efforts and findings related to compensatory 
education in general, ^e of which are relevant to subsequent discussions 
in our design, • (^ee Appendix I.) 

l'T'''^| Most research on the effects and the effectiveness 
^ of individualized instruction have been condu ct ed by groups or 
individuafs involved irr the development or implementation of the 
programs . 

With the exception of a number of doctoral dissertations 
and independent evaluations funded by groups such as /I/D/E/A/ 
Kettering, the majority of research on individualized programs has 

been conducted by. people involved in the development or implementa- 

> 

tion of a program. Although the standards of the research have 
generally been of high professional quality^ very few (if any) 
comparative studies of individualized programs (with the possible 
exception of SRI Follow Through Study) have been made. Hence, many' 
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^ of the findings below have limited gerreralizeability regarding 
"individualized instruction"/- — ^ 

* • 2, Siqnficant school environment or climatic chaffqes 
app'ear to be associated wi th the implementation of individualized 
instructional programs >' 

The relationship between school climate or environment 
and individual^ized instruction has been the re^arch focus of a large 

-number of studies sponsored by'/I/D/E/A/ Kettering'; however, such' 
evaluations have been limited to schools participating in the /I/D/E/A/ 
"IGE Network". In addition to "evavluations of the IGE program, the 
Wisconsin Research and Development Center for Cognitive Learning 
has evaluated and otherwise sponsored research on the Wisconsin 
Design Reading Program and other qarriculum developed by the Center 
(e.g./DNP anci IGM) , as well as IGE. Other studies tangentia>ly 

"focusing on this issue include the SRI F-ollow Through Study, and 
studies sponsored by Westinghouse Learning Corporation (regarding 
Project PLAN) by New Center. , * 

Summarizing the impafct of .individual ized instruction, 
particularly the Wisconsin design, Rossmiller ("Research and 
Evaluation Related to IGE", Draft, July 1975) summarized the - ^ 
findings of Gresso which are representative of o.ther studies of . " " 
IGE; ' ' 

"tligh-implementation IGE schools were signii^icantly 
more open-climated and famil.iar-climated than low 
•implementation schools. Low implementation IGE 
schools were significantly more closed-climated than 
high implementation schools." 
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I€E schools are character:ized byrthe schools having a greater 
degree of'openness^ the school't^ having more open autonomous climate, 
teachers having a greater degree. of esprit de' corps, 'pri^i pals having 
greaxef trust; principals havin^'greater consideration. in coiltrast 
to high-implementation IGE school's, loW-impTein^ntation schools are 
characterized by: the schools having a more paternalistic, closed 
cjimate; teachers 'having a higl\ degree of hindrance*; principal 
having a high degree of aloofness. ; * >^ - 

Other relevant finding^ include: a) multi-unit IGE schools' 

» « 
are more significantly achievement, change, and science oriented; 

b) teachers perceive school climate being more open in IGE s"c4iools . 

vs; non-IGE schools (Kelly, et al.);'C) that self-concept of students 

in IGE schools is significantly higher than the self-concept^ of 

students in non-^IGE schools (Nelson); d) that the environment of 

IGE schools was conducive to the development of favorable student 

attitudes towards peers (Edwards). As Rossmiller concludes: ^ 

"In general,, research fTas shown the climate of mu»lti- 
* unit schj«js|g>'^qibe^^ ^ different ' . /\ » 

from the NCfltftat^'^oVwfi^lra^tf^^^^^ schools . In 
terms of organizational^ cliliiate, multi-unit schools 
are mare open; are more change and achievement 
oriented; pos-sess staffs that are instructional ly 
s-atisfied, more motivated, and p^r'ceive greater 
levels ,of productivity; a,nd employ more educationally 
progressive practices*. ' In terms bf learning climate, 
f ' multi-unit school students have more positive self- 
concepts, more favorable attitudes towards fellow 
. pupils, and more favorable attitudes towards school." 

In terrtis of organizational patterns and decision-making at 
'the building level, findings from numerous studies including the 
TURNKEY Study of Compensatory Education in Michigan, indicate that 
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indivi dualized instructional programs'^are associated with; a) 
decentralized derision-making through a participatory process 
(TURNKEY, Walter, Smith, Gramenz); b) the role of the'principal 
as ,a facilitator, monitor, and leader (TURNKEY, Richardson'); c) 
a flexible and less rigid organizational structure allowing for 
shared decision-making and coordination (P^ligren, Herrick, and 
TURNKEY).^, . . ' ^ 

While the variables discussed above seem to be significantly 
related to the introduction and/or implementation of individualized 
instructional programs, other studies have made additional findings 
of interest tothis study. Kelley, in his study of the relation- 
ship between individualized instruction and school climate, found 
that a significantly stronger relationship between climate changes 
and the* introduction of individualized instruction occurs in rural 
and inner-city LEAs. In addition, he found that climate changes 
increased from the first to the second year; but in the third year 
tended to regress to a rigid, less open climate. Wh.ile open climates 
and flexible organizational structures characterize well-in^emented 
individuai-ized schools, Walter found that there existed stricter 
adherence to instructional accountability procedures in multi-unit' 
schools vs.- traditional ones, in addition, while greater role 
differentiation exists, school staff indicated a perceptibn of 
closer personal relations among staff. 

The results of the TURNKEY Michigan study indicates that a 
significant relationship between individualized instruction and 
school climate exists when compared with the school climate of 
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traditional or non-individualized programs. Many of these variables 

are outlined in the discussion of program variables in Section III 

of this report. 

3. Existing research- indicates that the introduction of 
— ' * — ■ — ■ J 

individualized Instruction has a significant positive impact upon ' 
student growth in math and reading (although the units of comparisons 
can, in many instances, be questioned,) 

, In 1973, USOE's Division of Compensatory Education conducted 
a synthesis of six major studies related to compensatory education: 

• Strategies of Compensation : A Review of Educational 
Projects for the Disadvantaged in the United States 
(Organization for Economic Cooperation and Develop- 
ment, 1971). 

• Compensatory Education : Evaluation in Perspective 
(Edmund W. Gordon, Information Retrieval Center on 
the Disadvantaged, 1970). 

• How Effective Is Schooling ? A Critical Review and Synthesis 
of Research F(|jdings (Final Report to the. President '3 
Commission on^cliool Finance, Rand Corporation, 1971). 

• ESEA Title I : A Reanalysis and Sy'nthesis of Evaulation 
' Data from Fiscal Year 1965 through 1^70 (American 

Institutes for Research, 1972). 

• Draft, Final Report, Exemplary Projects Studies (Columbia r 
University, 1972). 

• State Title I Evaluation Repdrts for FY 1972. 

In an attempt to identify common characteristics of effective 

programs, USOE identified several characteristics of individualized 
/ 

programs of instruction. As summarized in a NSPRA Report, these ^ 
characteristics included: 
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■ "Clear objectives which must be clearly written and stated 
in specific measurable terms; instructional techniques and 
materials must closely relate to those objectives. 



"Attention to individual needs which includes a ^careful 
diagnosis and individual plan for each student." 

"Flexibility and grouping which allow staff opportunities 
to provide small group instruction and to teach frequently 
on a on^-to-one basis. USOE notes' that when .group in- 
struction was part of the daily program it tended to be 
more effective if students were not confined to the same 
group for more than severa-1 days without reassessment of 
the teacher's and students '.strengths.-' 

• "Personnel management which allows key staff personnel' to ^ 
work individually with teachers in the classroom. USOE 
stresses the need for much coordination and cooperation 
among staff and a well designed inservice program." 

f "Structured program aporoach which stresses sequential 
order and activity. Pupils must also receive frequent 
and fimediate feedback." 

In 1973, Mayfield conducted a comparative study of conventional 
and individualized mathematics prograirts,- using the MAT to asses-s 
the growth of 40 matched pairs of students. While the student 
gains were not statistically significant overall, the individualized 
programs demonstrated si ightly higher gains in problem solving and 
l5computation. Significant gains in self-concept, measured by the 
Piers Self-Concept Scale, were found. Lazich's evaluation of -the 

« 

Wisconsin design for grades K-3, found statistically significant 
differences at the third-grade level in word attack skills when 
compared to a control group. A summary of the effectiveness of the 
Wisconsin design and other instructional" procedures and curricula 
developed by the Wisconsin Research and Development. Center synthesized 
the findings of more than fifteen studies. In all instances, student 
progress. in cognitive areas, as measured by "embedded" criterion- 
referenced tests built into the program, indicated student growth 
particularly in the area of word attack skills. In addition, 
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comparative studies' between the Wisconsin design and the Fountain 
Valley Teacher Support System and Prescriptive Reading Inventory 
indicated that students enrolled in the Wisconsin design program 
achieved significantly higher mean ratings, again measured by criterion 
tests. 

In Michigan* over the last three years, rather extensive 
evaluations have be'en conducted of individualized programs implemented 
under the rubric of "performance contracting'*". Based on norm- 
referenced standardized tests, student performance in cognitive 
areas in Menominee and Sault Ste^. Marie projects, involving five to 
ten buildings each, averaged 1,5 and 1.9 months gain per month in 
the program over the last three years, Menominee has developed an 
individualized mathematics program, while in Sault Ste. Marie, the 
reading program has been individualized. Only in the Menonimee^site was 

t 

showed only slightly higher gains in the experimental group (Bryan, 
1974), A review of the 50 projects approved by the Dissemination ' 
Review Panel (DRP) which utilizes "educational" gains as a major 
criterion, found that approximately one-third of the projects were 
individualized, with the remainder ranging from only slightly in- 
dividualized to relatively standard programs. In most instances 
the unit of comparison was expected growth, measured by norm- 
referenced standardized tests or normative comparisons by grade 
level. While control groups were used. for comparisons in a limited 
number of the evaluations, in no instance did the documentation 
reveal that random assignments to treatment and control groups had 
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been used. Indeed, as DRP projects were assessed by RMC as 
potential PIP sites, many DRP sites were dropped due to various 
external "threats'' to the evaluation designs, (Tatlmadge, 1975). 

4, Attempts to relate cost to the effectiveness in 
assessing individualized instruction find/or comparing it with 
"traditional instruction" have been minimal and simplistic 
' in design . 

/ The vast majority of the "cost-effectiveness" studies of 

individualized Education have been conducted or sponsored by model 

developers and publishers, or individual districts.. Th? findings 

are usually reported in a simplistic, cost-per-unit-gain, determining 

the cost associated with specific variables which cuntribute to 

performance* .Typical studies include: a) positive cost-effective- 
ly 

ness results in the Cincinnati Public Schools , through the use of 
the New Century .program; b) a cpmparative cos±-eftectiy,eness _ana lysis. 
Qf three types of individualized program in Dade County conducted 
by its Division of Research and Evaluation; c) a similar cost- 
effectiveness *study. of five types of reading programs in the Dallas 
Independent School System conducted ''by its Division of Research 
and Evaluation; and numerous studies by publishers such as Random 
House, Betti-Kit, McGraw-Hill, SRA, and others> Aside from their 
simplicity, for the most part these studies fail 'to capture all 
of the relevant resources, but rather focus on out-of-pocket or 
operating costs related to the specific program. 

In 1973 Boardman and Hudson developed a cost-analysis 
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niodel to 'identify cost factors directly associated with the im- / . 
plementation of IGE* The instrument for data collection consisted 
of self-reporting, personal interviews, and on-site visitations/ . 

r 

Similar to many of the evaluations conducted by publishers, this 

model ''attempted to identify the cost associated wi th implementing 

IGE,as a process ^(as opposed to the instructional programs used in 

* 

the I6E process); however, it did not attempt to^relate total cost 
or the cost of specific activities to any outcome measures. 

Several attempts over the last few years have been made 

* T 

to develop models to assess the cost .Sfld^ffectiveness of compensatory 
education programs. Since about 1971, the Education Testing Ser\;ice 
has been conducting a study of compensatory education reading 
prjograms to identify possible effects on the development ^of reading 
skflls in the elementary grades. 0ns phase of the study includes 
3 *cor!iprehensi*^/e evaluation of reading progran^ charact^rJstics-^d 
attempts to ana-lyze the cost-effectiveness of certain variables. 
Based on discussions with individuals 'involved in tbis study, 
several relevant issues are worth noting: 

• The definition of a program (i.e., similar studies receiving 
similar treatment) has created problems during the project's 
analysis phase because of the widewariety of "programs". 

• Success criteria and test administration procedures are 
being questioned in light of the occurrence of "bottoming" 
and "topping" affects, creating difficulty -tn assessing 
actual vs. expected gains. The model used fs similar 

to the RMC Model developed several years ago and uses 
standard pricing. It does not include a "trade-off" 
capability which is highly desirable for assessing 
individualized instruction^. A similar model is now 
being designed for use in the cost-analyses of Follow 
Through and the "Sustaining Effects" Study (USOE). 
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Since 1973, TURNKEY has been conducting a cost-effectiveness 
study for compensatory education programs for the state of Michigan. 
Although the program'did not foqus .solely on individualized instruc- 
tio.nar systems,' the results provide useful insights into the variables 
associated with effective programs and their respective cost-.* As ^ 
described in a subsequent section, this mode, with limited redesign, ^ 
appears ta be mgst appropriate for assessirfg the relative c^st-effecti ve 
ness of individualized and standardized -programs. During the ex- 
ploi'atory phase of the study, 45 variables were found to discriminate' 
.in a statistically significant m'anner between successful and 
'unsuccessful programs. In addition, over 30% of the variation -in 
student performance, as measured by standardized norm- referenced 
testing, was explained by per-pupil^osts of resources devoted to 
reading. The study identified a number of significant variables 

will (.11 nclc a^iUv- I u i-cu- uixn iiivjiiiUu>ai...-». f-..^.-.-. ii-^- 

a) amount of teacher tiTne devoted to diagnosis/prescription activities; 

b) occurrence' o'f teacher redesign/development of performance ob- 
jectives; c) diagnosing individual student needs as a major emphasis 
in pre-service training^; d) amount of teacher time spent in planning 
activities; and other instructional management variables.- As described 
in Exhibits 1-3 and 1-4, the cost associated with various , staff 
activities, and variables were also determined. The model used in 
Michigan, unlike others, attempted to determine th.e amount o# 
resources, regardless of funding source, consumed by an individual 
child throughout the year. The average cost in successful Title I 
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DOLLARS PER STUDENT FOR 



» » 


TWENTY-FIVE HIGH 

* • 


SITES , . 




* 


HIGH SITES ~ 25' 


FUNCTIONS 


AVERAGES 




COMP-Eb 


"f COMP-ED 






COMP-ED COMP-ED- 


COMP-ED , DECISION 


"admini- 


RESOURCE 


RESGIJRCES 


READING PLANNING 


TRAINING MAKING 


stration 


TOTAL 



PERSONNEL 

District Conp-Ed Director* $ — $ 17.84 $ 4.36. $ 12:80 

Principal • ^ .•— 15.08 . 7.8b 13.9,2 

Conp-Ed feacher * 193.60 53.52* 10. €4. " 30.^0 

Regular Teacher • . *^ 93.84 92.00 10.60* 49.80 

Paraprofessional 47.20 3.24 .80 liOO 

Reading Specialist V_^1.16 2.64" " .68 2.'24 

Other Classroom Staff'" , ' .24 — ^ 2.12 



$ 9.40 
9.20*, 



.84 



CONSUMABLES 

Comp-Ed Books and 
AV Software 
Regular Books and 
AV Software 

EQUIPMENT 

'Comp-Ed AV Equipment 
Other Comp-Ed Instructional 
Equipment 

Regular AV Equipment 
Other Instruction Equipment 
Comp-Ed Administration 
Equipment 

MISC^LLAIJEOUS 

Miscellaneous Comp-Ed 
Training Expenses 
Miscellaneous Comp-Ed 
Administrative Expenses 



24.40 
' 9.88 

2.76 
4.28 
3.72 



6.28 



.16 



3.16 



FUNCTION TOTAL 



$381.08 $184.32 



$41.36 $112.48 



$22.76 $742.00 
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Exhibit 1-4 






. DOLLARS PER STUDENT FOR 
TWENTY -THREE LOW SITES 








^ ^ HIGH SITES ~ 25 






FUNCTIONS 






AVERAGES 
RESOURCES 


COMP-ED 
RFADING 


COMP-ED 
PIANNING 


COMP-ED 
TRAINING 


COMP-ED 

DECISION 

MAKING 


COMP-ED 
ADMINI- 
STRATION 


RESOURCE 
TOTAL 


PERSONNEL 

District Comp-Ed Director 
principal ^ 

Comp-Ed '^Teachei' * * ^ 
Regular, Teacher 
Paraprof essional 
Reading Specialist 
Other Classrooni Staff 


•$.-'• 

135.96 
85.57 
70,26. 
6.04 


♦ 

$ 16'!S3 
i2.09 
17', 78 
65,48 
.78 
3.13 
. 2.83 


$ 4.22 
7,70 
2.83 
5.04 
.30 
1.57 . 


$17.48 
19.00 
11.78 
34.74 

:^2 

2.00 
.78 


W ° 

4^ ■^ •^ • J *m 

13.57 


$ 52.05. 
52. 36^^ 
168.35 
190.83 
72.56 
12.74 
3.61 


CONSUMABLES 

C^mp-Ed Books and 
AV Software - 
.Regular Books and 
AV Software 


a3.04 
9.87 


— 


— 






13.04 
9.87 


«^'' 

EQUIPMENT - 
, Comp-Ed AV Equipment 1.17 
Other Comp-Ed Instructional 
Equipment ^ .22 ^ 
I|egular AV Equipment 
Other Instructional Equipment 3.00 
Conp-Ed Adiainistraticn 
Equipment ' — 


•.a 


- 


— 


.39 


1.17 
. .22 » 
3.00 
.39 


; MISCELLANEOUS 

Miscellaneous Cotap-~Ed 
Training Expenses 
MisceLlaneous Comp-Ed 

'< Administrative Expenses 






■4.09 




• 

2.91 


4.09^ 
2.91 


[ FUNCTION TOTAL 


$325.13 


$118.92 


$25.75 


$87.00 


$30.39 


$587.19 






1-30 











reading programs of* $850 was significantly higher than conventional 
wisdom would indicate and suggests serious questioning of the 
$300 critical mass figure suggested in the WEW cost-effectiveness 
study of compensatory education (Lynn, 1972)* During, the cross- 
validation study preserrtly underway and scheduled for completion 
in November 1975, analysis will determine the instructional program 
characteristics, if any, which are associated with successful programs, 
Such.^n analysis should provide greater insights regarding the 
relationship between activities "inherent to individualized instruc- ' 
tion and program outcomes. In addition, as described in a sub- 
sequent section, the model will also provide information on the 
trade-offs associated with various types of instructional programs, 
the COST-ED model, modified for application in Michigan, is unique 
in its capacity for determining trade-offs and conducting sensitivity 
analyses. ^ 

In summary, very little is known about the^ effectiveness * 
and associated costs of factors contributing to the success of 
individualized systems. The concern expressed about the cost of 
individualized instruction (e.g., the RBS proposal) and' the ex- 
tremely low advertised costs of implementing individualized processes 
such as IGE (e.g., $10 per pupil) indicate a dire need to address 
this ,tssue in the design. , 

C. Implementation of Individualized Programs 

Existing p rocedures for assessing degrees of imp^ementat1on^ are in- 
adequate for the purposes of the study, thus requiring extterisive design/ 
deve1^pf)ment effort during the implementation phase . 
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While specific details regarding the issue of assessing - 
degrees of implementation are addressed in Section Ill/Task 3 
of this report,' based upon an analysis' of the SRI Follow Through instru- 
ment, the/I/D/£/A/ Implementation Kit, and various implementation instru- . 
ments (either observation or survey) used to assess various learning systems, 
the following findings in this area appear to be germane: a) that attempts 
to develop single instruments which would apply generally across all 
programs, such as the Follow Through instruments (SR^^jave resulted in 
findings of questionable validity; b) no^ existing instrument has been 
designed to assess degrees of implerp^^^ wel 1-implfemented 

individualized and standardized programs; c) existing observation and 
survey instruments to assess the degree of implementation of specific 
learning systems appear to be appropriate for redesign arid use. as a data 
collection instrument; and d) the implications for the analysis p^an ^ 
are significant indeed in that cross-program comparisons will be limited. 

D, Parental Involvement * 

Due to its research void and inherent sensitivity^ parental involvement 
can be treated as a progam variable only in a 1 imi ted. manner in this study . 
It should be addressed in detail in a separate study . 

In reviewing the minimal existing literature on the impact of parental 
involvement in compensatory education (Stearns) and based on observations 
and discussions with project staff, the following findings, described in 
greater detail in Appendix 2, have been made: a) there exists* a great void 
in research on parental involvement and its impact upon student performance 
in compensatory education, even through parental involvement is increasingly 
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becoming a requirement in most comp ed programs PB) the types of parental 
involvement, in terms of compensatory education generally, include advisory 
functions through Parent Advisory Committees, as employees providing 
assistance to teachers in the classroom, as tutors or instructors in at- 
home programs or components, and as participants in the establishment of 
individual objectives; c) the quality of parental involvement in the above 
categories varies considerably, requiring the development and use of 
instruments which will capture qualitative differences amf contextual 
situations. Based on the above findings, we are proposing a separate 
study, as described in Appendix 3, 



In thi? section of our'Report, we have .described the background 
of individualized instruction in compensatory education programs 
and .discussed the staie of the research art as it relates to the 
^-purposes of this study. 
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SECTION i I ' • 
GENERAL APPROACH/-STUDY DESIGN 

In this section we discuss the general approach which we are proposing, 
the majoV design issues and technical approaches taken in each of the tasks 
outlined in the original RFP and our general^ philosophy which permeates 
the proposed evaluation design. Where appropriate, to reduce redundancy, 
we reference prior sections both in this report and in our initial 
Proposal, in discussing justifications and rationales. 

MAJOR STUDY OBJECTIVES / , 

Based up6n the general objectives specified in the RFP and in light 
af the findings' Which* arose during the de$ign*phase, we are proposing the 
following major study objectives: ' ^ * ^ , 

1, To determine the differential effects and effectiveness of well- 
implemented individualized vs. well -implemented standardized 

. math and reaching programs, which provide instruction for compensatory 
education student?; at the elementary level. More specifically: 

to evaluate *the relative impact on cognitive growth in math - 
; and reading; 

• ^ 'f to compare the. impact upon student and. teacher attitudes and 
; student outcomes in the affeptive domain;^and 

\^ d to identify the relationships among indi vidualfzat'ion,^ class- 
room environment, and schopl climate^ identifying' and ^na.lyzing 
possible reasons for these relationships 

2. To determine the relative cost-effectiveness of the process 
dimensions of individualized ins-tructjon/. ' * - 

3i To determine the degrees to which potentially v/ell -implemented 
programs are implemented during ^the period of observation and 
in turn, the relationship between degrees 'of ifnplementation and 
program' outtoraes within and among programs. 
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SAMPLING PLAN 



The sampling design which we ar^e proposing requires data from 



approximately 800 classrooms with equal or proportional representation 
in each of the four categories (I, II, III, and IV) illustrated in Exhibit 
II-l. Equal or proportional representation within math/r.eading and by 
malnstream/pull-Dut classroom settings will be maintained. An equal 
number of classrooms from grades 2 and 3 are specified at each substrata 
of this framework. Potential candidate sites have been identified (or 
will have been identified) through District Survey I in accordance with 
its initial data collection effort and through the efforts of the project 
team, the latter including selection from documented and validated well- 
implemented programs, nominations of well -implemented programs from p.ublishers 
and/or model developers, and TURHKEY and other documentation sources. 
During the design phase, "potential" candidates were verified to be 
"likely" candidates through telephon.e conversations and extensive documen- 
tation review, to ensure the feasibility of the sampling plan. During 
the implementation phase, further verification of the sampling plan through 
questionnaires and on-site visitations will be required^prior to final 
selectipn of participants. 



Based upon a review of relevant literature and research findings, 
including studies conducted by TURNKEY, and, In accordance with our sampling 



the proposed "planned variation" study: a) type of program, according to 
the degree of individualizatilbn; b) instructional setting (e.g., mainstream vs. 



VARIABLES 



plairr we have identified three categories of independent variables in 
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Exhibit II-l 



BASIC SAMPLING b"RANeVORK 







Specific Perfoniiance Objectives 
are assigned to each student 


Specific Perfomance Objectives 
are not assigned to each student 




Diagnosis/ 
Prescription 
is continuous 


Diagnosis/ 
Prescription 'is 
not continuous 


Diagnosis/ 
Prescription 
is continuous 


Diagnosis/ . 
Prescription is 
'not continuous 


Individual Learner or Material Path? 

exist 


Individual Pacing 
exists. 


I 


II j 




Individual Pacing 
does not exist 


^ t 

II 


9 

* 


1 


Individual Learner or Material Paths 
do not exist 


Individual Pacing 
exists 




III 

^ ^ — ^ 


Individual pacing ' 
does not exist 




1 r 

'in 


IV 
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pull-out); and c) subject area (i.e,, math or reading). In addition to 
a number of control variables, including: a) student achievement in math 
and reading; b) classroom environment and climatic variables; and c] 
affective variables relating to students and teachers, including satisfac- 
tion. In addition, we have identified a number of implementation variables 
which will also be used as program variables to relate to various outcome 
variables. ■ . " / 

'INSTRUMENTATldr 

The instruments to be utilized in the proposed des'ign include: a) a 
limited number of existing instruments; b) existng instruments which will 
necessarily require modification; and c) newly-developed instruments. 

In order to assess student outcome measures in math and reading', 
we are proposing to utilize the Stanford Achievement Test (Primary I, II, 
and III) at grades 2 and J. If after program selection it is determined----^ 
that the objectives and content of the programs differ significantly, we 
propose to use selected items from the SAT as objective- referenced or criterion- 
referenced tests to ensure program fairness. A unique set of procedures 
based on the work of Klein is'propos'ed. 

For interview purposes, we are {)roposing to modify the instruments 
for interviewing staff used in the Michigan study of compensatory Wucation 
conducted by TURNKEY. For the purposes of measuring student outcomes in 
the affective domain, we are proposing that the revised Piers-ilarris 
cKildren's Self-Concept Scale (recently validated in^an inner-city school 
for grades 1-3 by the Purdue Education Research Center under a USOE con-^ 
tract). be used along with the Attitude Towards School to assess student 
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attitudes towards school; the Organizational Climate Index v/ill be used 
to measure teachers' assessment of classroom environment. Individual 
instruments following a uniform format are proposed to assess degrees 

of implementation in the form of checklists and/or observer vjerifica- 

tion forms. As an option, we'have included an instrument which could be 

♦ 

used for assessing implementiofTof the dimensions of individualization, 
if this instrument is found to. be feasible during field testing. 

DATA COU^GTlOiN - • * • , - 

i 

The data collection plan which we proposed for project implementation' 
consists of thVee types of data collection procedures: 

• Classroom observations, (including verification); 

• Staff interviews (bath structured and open-ended); and ' 

• testing. 

Observations of compensatory education classrooms will occur periodically 
during the 1976-77 school year in all classrooms. Interviews of principals, 
teachers, aides, and other appropriate staff members will be conducted early 
in the school year and followed-up with parallel verification interviews 
in the spring of 1977. Achievement and other tests will be administered 
as early as possible durng the 1976-77 school year as a pre-test and again 
later in the spring as a post-test, as prescribed in the SAT Manual. 

The data collection itself will be supervised and conducted by local 
personnel and will be coordinated by a regional structure of core staff 
members. These data collectors will be experienced professionals who 
will "have familiarity with local policies, procedures, and personnel. 

If) 
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Through the use of local personnel rather than outside data collectors, 

the twin problems of data confidentiality and site cooperation iT)k^e more 

V J 

readily addressed^ * ' ' 

DATA REDUQJION AND ANALYSIS ^ ^ 

Data obtained throughout the 1975-77 school year from pre-test 
administration and from periodic observation of program operation and 
characteristics will be reduced for analysis upon collection. Procedures 
for the speedy reduction of the post-test results and other late school 
year data will be carefully refined during this school year to en5ure that, 
once the data is obtained in May 1977, all necessary data will be reduced 
and analyzed in time to report the study* s results in July 1977. Since 
TURNKEY experience has indicated the extremely critical nature of these 
data reduction task§, sufficient resources will be allocated to this 
activity in our proposed design to ensure their timely completion. 

Once all data has been obtained^ the carefully planned data analysis * 
effort will proceed. This effort will address all research questions to 
be covered by the study in a specific sequence. These questions are \ 

hierarchical' in nature for the proposed study, reflecting the inter- 
ac.tion effects nature of our semi-factorial study design. , The techniques 
to be applied in our proposed series of control contrast range from 
simple comparison of means or frequencies via tests or contingency table 
methods to more sophisticated methods such as ANOVA discriminant function 
analysis. 

UNDERLYING PHILOSOPHY OF EVALUATION 

Reflected in the above general approach and implicit in our discussion 
of how we propose to conduct specific tasks is an underlying philosophy 
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about evaluation ^nerally anu about federally-sponsored evaluation 
studies/experiments in education in part^icuTar, While we are cognizant 
of recent literature regarding studies of policy research in education, 
one's own experience as a participant and/or outside observer of large- 
scale study efforts similar to this onq, necessarily tends to influence 
one's approach. A brief understanding of-this philosophy will perhaps 
put some of the subsequent discussion in a propoer perspective. 

First, we believe. that the ultimate purpose of evaluation is to 
provide useful and timely information to decision-makers for the purpose 
of improving programs, more so 'than proving whether or not a program works% 

«» 

The "users" of information resulting from this study will be Congress 
as they formulate education policy and, ultimately, local education agencies 
as they have the prime responsibility for ensuring improvement of the 
operations of compensatory education programs. Proving that a particular 
program works is less interesting than determining why and under what 
conditions. Moreover, when the perceived purpose of evaluation is to' 
"improve" than than "prove", the negative connotations sometimes directed. 
to\>(ard recent federal evaluations can be minimized. 

Second, while v/e realize that social science research in general 
and, specifically research related to education, is highly challenging, . 
we also feel strongly that the federal role in evaluation presents 
an awesome responsibility. We applaud attempts at the federal level to 
improve the state of the art in social science research. However, as the 
federal government experiments with new evaluation and experimental techniques 
one has to realize that education officials at the SEA and LEA levels 
are more concerned with "reported results" than the advent of new techniques. 
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In the^ recent past aTlegations have been made that one of the reasons why ' 
^''commonly accepted successful programs" did not fare favorably in national 
evaluations was due to the inadequacy of the evaluation designs specifically 
and the inability of the Government (for inter-governmental, political 
and other reasons) to conduct experiments/evaluations effectively. In 
our proposed design we have attempted to strike a balance between proposing 
techniques and instruments which we feel are technically sound and appropriate 
(although they may be non-traditional), wh^ile at the same time seeking to 
minimize the probability of questionable findings due to research flaws. 

Third, we feel that large-scale federal evaluations and study designs 
should.be flexible and -accomodati ng rather than rigid and parochial , 
taking into account the political, social, and economic milieu in which 
education actually occurs in this country. For example, while true 
and pure experimental designs may be relevant^ for disciplinary research, 
ahe feasibility of applying such designs in policy research of this 
nature may create the need to gravitate towards "quasi-experimental" 
designs. Similarly, in data collection one might have to opt for instru-' 
ments and procedures based .upon a trade-off between what is technically 

» 

-possible ancj.what can be accomodated by the LEAs and programs being studied 
to ensure a modicum of cooperation necessary for the completion of any 
study of this kind. We feel this is particularly true^n this study 
where most of the programs studied will have been developed and/or operated 
through resources which, for the most part, are locally generated and/or 
allow Vocal discretion. 

Fourth, federal evaluation techni^ques and research desigr^s should be 
appropriate and should ensure "fairness". Too often the criteria used to 
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judge the effectiveness of programs at the LEA level differ significantly 
in terms pf priority rankings ^rom those at the federal level, Care should 
be taken to insure that success criteria are generally accepted and where 
divergent criteria exists, the design should accomodate a number of them. 
Our proposed design reflects this consideration. Moreover, evaluation 
techniques should be program-fair, assessing the progress achieved in 
programs in accordance with intended purposes. For this reason, we have 
taken great care and suggest a considerable amount of time to be devoted 
to ensuring that instruments, particularly implementation instruments, 
do in fact assess what is supposed to happen at the classroom level. While 
many tenets of the above philosophy have been espoused before, we 
propose that the implementation contractor use the above principles as 
a guide in implementing the proposed design. 

* ♦ ♦ 

I 
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SECTION III 



DESIGN ISSUES/TASK APPROACHES 



In this section of our report we outline the specific design issues 
and approaches to the first five tasks specified in the RFP. Note that 
the tasks we refer to here are listed as subtasks of the overall study 
design task on page 20 of Enclosure I in the RFP. We have adopted the 
notation of nine "tasks" rather than nine "subtasks" within an overall 
design "task" simply for convenience. This section of our report 



concentrates on the five "design" tasks (as opposed \to "data collection," 




which are discussed in Section IV). 
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TASK 1 DEFINE PROGRAM VARIABLES 
Design Issues 

The sampling, control, and additional descriptive^variables of 
interest to the study must be carefully and rigorously determined since 
these definitions will affect all other aspects of the study. Sampling 
procedures, interview and observation schedules, the data analysis plan, 
and the conduct of data collection efforts all depend, ''for their 
framework, upon the definitions of prografli variables arrived at in this 
task. "The ability of NIE and the implementation contractor to assert 
that they assessed the effectiveness of individualized instruction in 
reading and mathematics programs for compensatory students will depend 
in large part upon the quality of these definitions. Ultimately, 
Congressional use of the results will also depend upon the utility of 
these definitions and the quality of the proposed design. 

As indicated in -the RFP, a crucial aspect in defining the program 
variabl-es will be the identification of operational criteria of what is 
an "individualized" program or a "standardized" program. TheJURNKEY 
team recognizes that the.crite^, as listed in the RFP, are basic: 

-J • . individualized programs will have stated or written performance 
objectives prior to implementation of the instructional 
strategies relied upon to meet these objectives; 

i individualized programs will rely upon diagnostic procedures 
for assessing the instructional need? of each individual 
student relative to the stated objectives; 
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J # individual programs will provide instruction to each individual 
child that is a prescriptive reflection of the diagnosis of 
his needs relative to the stated objectives; 
* * » 

• individualized programs will rely upon progress tests in 
order to assess mastery at regular intervals during the 
program; and 

# individual izeji programs will be characterized by individually 
paced instruction ahd performance-based student progress and 
instruction tied to the student's results on progress tests. 



This list of characteristics must, however, be operational ized further 
for a number of reasons. . - . 

At this time a large number (to be increased in 1976 school year 
because of new Title I guidelines) of compensatory education programs 
operate wrth a set of stated or written performance objectives, although 
a sizeable variation may exist among the qua'lity -and appropriateness 
of these objectives. The results of our recent study of compensatory " 
education practices/in Michigan indicate that 71% of the compensatory 
education teachers sampled from the most "successful" districts stated 
that they provided reading instruction that was d/rectly related to a- 
s'et of written performajice objectives. Similar teachers for the "un- 
successful" districts indij;ated that 63% of their number also provided 
. irrstruction related to such objectives. Moreover, where these objectives 

>r 

apparently existed, 80% of the^respondents claimed that the objectives 

were tied to individual student performance rather than group perfor- 
mance — for both successful and unsuccessful programs. In the Michigan 
study design, programs were not selected on a scale of individualized vs. 
standardized, but rather on a scale of program success. From this sample* 
of Michigan programs, however, it appears that nearly 70% of Michigan 
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compensatory education programs were operating with such written.,objectives. 
Since it is unlikely that those districts, which re^y upon writt:en 
objectives, alT provide completely individualized programs, it is 'apparent 
that while th^ existence of such objectives is necessary, it is certainly 
not a sufficient condition to allow a program , to be labelled "individualized 

Further,, it is also^^-^rue that many teachers rely upon some typfe of 
diagnostic testing, especially for compensatory programs where Federal 
guidelines set forth such a procedure. The tests relied upon for diag- 
nostic testing may be specially developed or standardized or may even be 
subjective judgments; but, whatever their form, such tests are like 
written objectives — so common today that their existence is only a 
potential indicator of individualization, not a guar<ihtee. 

Similar concerns may be raised relative to prescription-based 
instruction and the use of progress tests'" to assess mastery. -However, 
progress tests may not be as commonly used as diagnoses/prescription and 
may, therefore, be a better measure of individualization than the others . 
so far discussed. 

Even the key element of individual-pacing, where instruction is tied 
to measured student progress, must be examined critically to assess the 
uniqueness of these characteristics to individualize2f'instruction. In 
'the Michigan study, teachers in the successful sites were asked what 
percent of their students' reading time was spent in individually-paced 
activities; their average response was approximately 85%. 'Similar teacher 
in tt^e unsuccessful sites responded with an average of about 74%. 
Thus, the aspect of individual pacing was fairly common' to all programs 
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studied, indicating that individual pacing (or teachers* perceptions of 
individual pacing) is not necessarily synonomous with individualized 
instruction or that individualized instruction was not a unique characteristic 
of successful programs. 

Ultimately, it might be ^that the key characteristics of an educational 
program that would allow that program to be irjcontrovertibly labelled 
"individualized" is the indiyiduaUzation of the program's response (in 
terms of material; content, techniques, etc.) to the results of the progress 
checks. For instance, if all learners in a programuse the same materials 
but some, are allowed to use them at a slower pace than others, then it 
is qui^fe possible even with all of the other accoutrements of indivi- 
dualization in force — that this program i^ only a time-eased v.ersion of 
our traditional programs. But in-order for a truly individualized instruc- 
tion plan or set of- instructional events to be carried out in rcsporso to 
each individual child's progress test results, then each of the preceding 
elements (objectives, diagnosis, -prescription, progress evaluation) must 
be part of the overall program. The potential for self-pacing is inherent 
in the existence of multi-paths for the instructional/pla/l to follow/ 
Thus, although the existence of these multi -paths might be necessary 
and sufficient in itself to justify a Isabel of "individualized", the ' 
existence of these other aspects is" necessary to validate any claims of 
effective individualization. ' . ' ..^ ' ' 

While this discussion of compensatory education as actually practiced^ 
in classrooms was intended to illustrate the difficulty in contrasting 
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programs labelled "individualized" with those labelled '^standardized", 
by defining a "standardized" program as one which contains none of the 
elements just discussed one would probably find only-a very few such, 
programs within the universe of Federally-supported compensatory education 
programs.- Siven the moderating effects of LEA and SEA regulations and 
traciiTrons" (e' g7roYer^T4"3E?VrifiaMaTe~ vari^u^ 

objectives) defining an ."individual ized" program as one which completely 

embodi.es all of these aspects to a maximum degree would also result 

in the identification of only a very few such programs. Even if such 
♦ 

extreme cases could be found, it is clear that policy research (consider- 
ing Congress as .its prime audience) should not be greatly concerned with 
such unrepresentative cases. ^ 

In our Prcfposal (p. III-7) we suggested a two-staged activity to 
operationally define individual ized and standardized programs during the 
design phase. To summarize, five questions relating to: sequence,^ pacing 
use of individual progress checks^ use of diagnosis and prescription, and 
documentation of writt-en performance objectives would be asked and scaled 
on a continuum indicating the degree of individualization (e.g., number 
of Is vs, S'S), The second step as originally proposed would include^ 
a review of documentation to be gathered during, the implementation phase o 
the contract to validate the' initial responses. As described previously, 
the project team collected documentation on a large number of projects 
and conducted a second stage activity for many sites during the design 
phase. In addition, a limited number of observations were conducted in 



conjunction with other projects and activities involving members of the 
project team* The net results of this "field testing" of the approach 
resulted in the final proposed definitional framework which has been 
included in our sample design* 

The definitional matrix will be used to tlassify whether a program 
meets the following characteristics: 

« 

"PO'V Whether specific wntten (documented) performance 
objectives are assigned to individual students. 
Objectives would also' include stated/cormiorily known 
criterion or proficiency levels indicating mastery 
(paper and pencil or pprf ormance-based.) . "Givews" and 
"time periods" would be consideri&d desirable but not 
necessary. 

"D/P" Whether initial placement H'or each student in the program's 
materials is based on a diagnostic test and_ a specific 
process is subsequently followed for diagnosing student 
. needs and assigning prescriptive materials or exercises 
on a continuing basis. By 1976, all Title I schools 
will be required by regulation to administer "initial" 
or "placement" diagnostic tests to all students; Hence, 
the -emphasis op c ontinuous diagnosis . 

"ALP" Whether uniquely prescribed individual learner paths 
through the relevant program materials, related to 
performance on prior activities, are followed by each 
sludent. Preliminary review of project documentation and 
discussions with LEA staff i ndicated^ that in a limited 
number, of programs which were alv/ays\described as being 
very "individualized", in all instances individual 
students 'participated as part of a smal\l group for 
relatively short periods of time in conducting specific 
exercis^es or\drills. Hence, an indiVidusN learning path 
which include^ some participation in smallXqroups or 
specific activities would be allowed. This ^dimension 
is, in fact, an^xt^nsion of the prescr1ption\tharacteristic 

"PACE" Whether or not the amount cJf time required for W student 
to master a specif ic\performance objective or to complete 
a given portion of a program' is determined individu^^lly 
for>^r by that student and varies from student to stbdent. 
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Program Variables 

The RFP asked the design contractor to identify program operating 
characteristics of interest, control, and descriptive variables. In our 
Proposal we discussed a number of categories of descriptive variables 
used by TURNKEY in its study of compensatory educfation programs in 
Michigan. Specifically, nine groups of variables were hypothesized as^ 
acting together in their impact on program effectiveness. Through the 
application of various statistical techniques, 45 variables were found 
^to discriminate statisticjally be'tween successful and unsuccessful programs. 
This study is continuing and, at present time, is in a cross validation 
phase with expected results in November 1975. The specific variables 
which were significant during the exploratory phase of the study are 
included below as part of the overall list cf variables and specifically 
in Appendix 4. In ad'dition, as noted in the prior section of this 
document an extensive review of recent evaluations relating to this 
study was conducted from which additional program variables were identified. 
The results of our efforts during the design phase thus far are listed 
below in terms of the various types of categories of variables. It 
should be noted that implementation variables are discussed in Task 3. 

The variables to be studied by means of the proposed study design 
fall into three major categories: Independent, Moderator, and Dependent. 
Each of these categories is described below and th'e specific items to be 
studied within each item listed and the rationale behind the inclusion ' * 
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A. Independent Variables 

The variabi^Jncluded within this category are those program 
factors whose impact we wish to investigate directly by incorporating 
them into the (stratified) sampling plan for the study^ We accomplish 

this by means of a specific sampling plan which follows an analytical 

^ 

structure designed to address these variables explicitly^ There may 
be other program factors whose impact we v/ould wish to study, but only 
those reflected in our sampling design can be viewed as independent 
variables^ The program factors included in this category are Itsted in 
Exhii)it III-l along with the suggested data collection method and the 
rationale for the inclusion of each item. 

B. Moderator Variables 

The variables included within this category are those program 
or student factors whose impact we wish to control by holding their 
values constant or to study as covariables* wi th our dependent variables. 
The first of these two types of moderator variables is referred to as 
Constants, and the second as Oovariables. Each of these two groups 
is described in further detail below. Although the covariables are' not 
reflected directly in the sampling plan, their impact on the dependent 
variables CSn be assessed statistically by incorporating these variables 
as jqovariates In analyses of covariance. 
1 . Constants 

These are program or student characteristics whose impact we 
wish to hold constant for all analyses which are con,ducted .as part 
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INDEPENDENT VARIABLES 



ITEM 



SUGGESTED DATA COLLECTION 
METHOD 
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RATIONALE 



Type of program as categorized 
via definitional matrix (see 
Task 2) 

>,e.5 Whether a program is 
very individualized (Group I) 
individualized (Group II) 
standardized (Group III) 
very standardized (Group IV) 



Instructional setting of 
specific comp ed'' instruction; 
specifically v/hether the 
program oper^te$ in a main- 
stream or a pull-out fashion. 



Subject area: specifically 
reading and math. 



Sampling data (see Sampling * 
Plan) verified during on-site 
visit to specific classrooms 
plus defined in more detail 
via on-site observations in 
specific classrooms, Observa- 
tions/Survey items to include: 
0 student record folders to 
note time required to 
•complete units or materials; 
t identif icati-On of fraction 
of core materials which are 
used by some but not by all 
stiHents ; 
t the diagnosis/prescription 
process ; 

• identification of objectives 
associated with specific 
students including the 
ej<;tent to which students 
can describe the objective 
currently being addressed. 

Sampling data (see Sampling 
Plan) verified during on-site 
visits to specific study 
schools. Data required would 
include a description of what 
portion of the instructional 
program is provided in what 
setting for the comp ed students 

Sampling data (see Sampling 
Plan). 
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of this design. In reality," these constants may have'an impact 
on the dependent variables; but for the reasons detailed in Exhibit 
III-2, we choosjg tq view their impact as being beyond the scope 
of this study.. These reasons are listed in Exhibit III-2 along 
with other information related to each item in this category in 
a manner sjmilar to Exhibit III-l . 
2. Cbvariables 

These are program factors or student characteristics whose 
impact we wish to study (as is the case for the independent 
variables). However, we have not specifically reflected a balanced 
and ordered sample of relevant levels of each of these blocking 
variables in our sampling plan, a process which was specifically 
followed for the independent variables. Thus, the covariables are 
measured in a manner similar to .the independent variables, and 
their impact may be studied only to the extent that the daita 
resulting from the sampling design provides a reasonably represen- 
tative distribution of values for the covariables. The program 
factors and student characteristics included in this category are 
listed in Exhibit III-3 along with the suggested data collection 
method and the rationale for the inclusion of each item. 

C. Dependent Variables 

the variables included within this category are those student 
school characteristics we are usir\g as the measures of effectivenes 
the j)roposed study design. These measures fall into two major areas 
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CONSTATS 
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Grade level all programs^ 
•studied will be operational at 
grades 2 and 3 and classrooms 
for each of these levels will 
be equally represented* While 
grade level comparisons 
between these two levels will 
be possible, U is felt that ' 
by studying these contiguous 
levels we can best understand 
the impact of individualization 
— a concept in which grade 
level becomes less meaningful 
beyond adrriinistrati ve nsGds. 


Sampling data (see Sampling 
Plan), verified on-site. 
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All students will be 
"corapensatory education 
. students", i.e., either Title I 
or title I eligible (.it is 
felt that this wider 
definition is needed to iden- 
tify sufficient Group III and 
IV programs).' 


Sampling data (see Sampling 
Plan), verified on-site. 


X 


X 




X 


All programs will have been 
operational for at least the 
two previous years in the 
LEA,, and at least the previous 
year in the subject school 
(for stability purposes and 
to help ensure "well 
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Sampling data (see Sampling 
Plan), verified on-site. 




X 


X 


X . 


Subject students as much as 
possible be all native 
speakers of English at home 
(to minimize testing problems). 


Sampling data (see Sampling 
Plgn), verified on-site. 
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CONSTANTS (Continued) 
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ITEM 



SUGGESTED DATA COLLECTION 

METHOD 
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All programs will be "well- 
implemented", whether 
individualized or not (design 
constraint) . 



Sampling data (see Sampling 
Plan), specifically the 
implementation/effectiveness 
question, the stability items 
^programs, teachers, students), 
"the documentation items. 
During the study, implementa- 
tion to be addressed via 
observation of actual 
'definitional matrix category 
vs. i-nitially assessed 
category, controls placed on 
testing scoring procedures to 
reflect actual instructional 
content in each program, and 
observer judgment of the 
degree to which the program 
seems well run according to a 
list of general guidelines for 
well-run programs that would 
be developed by the study 
contractor. 



ERIC 



G2 

III-13 



EXHIBIT III-3 



COVARIABLES 



ITEM 



Number of years that the 
subject Student (Ss) have 
been in the program; 
specifically one or more 
than one. 

Gender of ^s. 

Parental involvement in 
program. 



Class size, 



Cost of the resources 
consumeii in the program, both 
during instruction and in 
support activities. 

Pre-test scores (used to form 
residualized gain scores)-; 

Presence/absence of relevant 
instruction in content area 
addressed by test. 
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SUGGESTED DATA CflLLECTION 
^ METHOD''^ 



Sampling data where possible 
^.verified during on-site 
visits; but mainly ^through 
survey data gathered during 
the study year. 

Survey data gathered during 
study ye'^r.' ^ 

Observation of parental class- 
room activities, if any, 
during the study year"; mainly 
through survey data gathered 
during the study year. 

Survey and observation data 
gathered during the study year. 

Survey data from TURNKEY'S 
cost-effectiveness instru- 
ments (see Dependent 
Variables - Process below). 

Pre-test score results for 
sample Ss . 

Teacher survey; work or student 
progress test samples; 
examination of'existing 
performance objectives. 



RATIONALE 



cx: 



UJ UJ 

o en 



CO 



O 
» — < 

< 

2: 
UJ 
2: 
UJ 



III-14 



63 . 



outcome variables and other associated variables. Under the area of 
outcome variables, measures in both the academic (cognitive growth) 
and the affective areas, are included. Under the area of other associated 
variables, measures in the area of classroom environment, as well as 
a number of factors to be studied post tioc (i.e., factors which are not 
reflected in the sampling design) are included. Each of these areas is 
described in further detail below: 

1 . Academic Variables 

These are measures v^hich encompass cognitive growth in the' 
.basic skills of reading and math, as well as other such school- 
based measures of student performance as attendance and behavior 
patterns. Exhib'it III-4 lists these factors. 

2. Affective Variables 

These are measures which encompass the affective or 
attitudinal domain of student and school characteristics. 
Exhibit III-5 listb these factors. 

3. Classroom Environment Variables 

, These are measures which encompass general j tems related to 
classroom environment as well as specific factors which we wish 
to study after the^fact. Exhibit III-6 lists these factors. 
Appendix 4 lists the other factors suggested for study as part 
of this category of variables. The items included in Appendix 4 
are derived from TURNKEY'S existing process evaluation instruments, 
which have*bQen applied in numer^ous, program studie^s throughout the 
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EXHIBIT JII-4 



ACADEMIC VARIABLES 



ITEM 



SUGGESTED DATA COLLECTION 



RATIONALE 
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Residual 1 zed achievement gain 
scores (with pre- test co- 
variable effects removed^. 



Mastery of objectives 



Scores from standardized test 
or sub scores (perhaps 
specially defined) from such 
tests, matched in coverage to 
the content areas covered in 
instruction . 

Scores frpm criterion- 
referenced tests administered 
for this study, matched in 
coverage to the content areas 
covered in instruction 
(post-test only) . 
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Student self-concept: 


Existing instruments including: 
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feel self 


Self Concept Ability Scale 














school self 


Piers-Hams Children's Self 














behaving self 


Concept Scale* 














Student's attitude tovvard: 


Existing instruments including: 
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school work 


Attitude Toward School* 














tgacher 


My Class Inventory 














.school in general 


School Attitude Test: Oral 
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School Sentiment Index 














Attendance 


Survey of school records 
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*Instrunient v/hich is recommended for use. 
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CLASSROOM ENVIRONMENT VARIABLES 
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staff assessment of class- 


Existing instruments including: 
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room environment: 


Class Activity Questionnaire 












intellectual climate 


Learning Environment Inventory 












achievement standards 


Minnesota Teacher Attitude 












organize effectiveness 


Inventory 












supportiveness 


Organizational Climate 












ord^rli ness 


Descriptive Questionnaire 












impulse control 


Organizational Climate Index* 














Purdue Teacher Opinionnaire 














Social Climate Scale 












Factors related to classroom 


Existing instruments: 












and program process management 


TURNKEY'S Process Evaluation 












data including: 


Instruments and 












cost effectiveness 
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(See Appendix for a full 
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list of relevant variables.) 
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^Instrument which is recommended for use. 
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country. Listed'in Exhibit Iir-6 for illustrative purposes are a 
V number of process asp^ets-fourid to be critical for effec^iive 

programs in TURNKEY'S work to date. 
' The analysis of the relationships between the previously listed 
items and the other factors suggested here is recognized as being one that 
XDTmhT^3iJtt-iTr^ — 
•important variables, if only because the number of items listed in 
Appendix 4 is quite large and a small percent (such as a chance rate of. 
5%) of a very large number is still a large number. Thus, care must be 
exercised in stati^ the implication of any results in the area. It is 
also recognized that such an analysis, even if identifying a truly 
significant number of important variables, would' at ^best on]y form the 
basi^ for subsequent cross-validation studies that would control fQ,n the 
identi fied- factors in a luore systematic manner, rlowever, since TURNKEY - 



has already developed results from applications of the TURNKEY process 
evaluation instrumeirts in a sizeable number of programs, it would be 
possible to use the existing" TURNKEY data to predict the results from 
the NIE data base. Thus, the data even for this one year of study 
(1976-77) could serve to cross-validate the TURNKEY results to date by 
evaluating the success. of each of these predictions. 

Note that- in Exh.ibit II 1-6 we have indicated that some of the items 
contained in the TURNKEY survey instruments mentioned there will be developed 
into an observation format. The observations obtained'^in this format'would 
be collected along^wi th^ the program implementation observations (see Task 3) 
and used to verify survey responses^ in these subareas obtained during early 
data collection interviews. , , ^ 
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TASK 2 -DEFINE SAMPLING PROCEDURES • ' • 

' The procedures tV be utilized in the determination of the sample of- 
district's, schools, and classrooms to be included in the study fall into 
four principal subareas: 

• the determination of' the sampling frame/definitional matrix; 

• the development of sampling criteria (i.e., for both degree 
of individualization and degree of implementation); 



..the sample selection procedures; and 
final .sample dimensions. 

As was indicated in our Proposal, the selection of 'programs must include 
considerations of ' both degrfee of individualization and degree of implemen- 
tation. -The first of these considerations, would be estimated for any 
given- program on a .priori basis by a reviiew of existing program documentation 
. and tentativ^ conf-irmatipn. - The sec6nd consideration requires ^^^*<^"9h 
• analysis -of program documentation and verification of activities (see 
,Tasl< 3.)-. Our study design includes provisions for treating programs 
which, upon closer scrutiny during project implementation, are found to be 
:-less "well -implemented" than originally estimated/ 

In this sec?tion, we "describe the' recommendations, and rationales for 
■ each of the four subareas which comprise the ov/rall sample sei^ction process. 




The Sampling' Frame/Defi pi tional Matrix 

The framework within which both the sampling and analysis i#ocesses 
must fit is' determined'by three dimensions: grade level, instructional 
setting, aod type of program ('degree of individualization). 
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A, Grade Level 

The proposed study is' to consider programs operated for comp ed 
students at grade levels K through 4* We recommend that only grades 
2 and 3 be specifically examined within any program studied. We make 
this-recommendation for the following reasons: 

• It seeThs unnecessary to study programs at all five grade levels 
(e.g., for reasons used in the SRI Fol low Through Study); yet 
then'ntended study must be beneficial at the lower elementary . 

' level. A subset of two of these five grade levels, properly 
selected, would suffice for this purpose. 

• Any national study which ultimately must entail the use, in . 
some manner, of standardized norm-referenced achievement tests 
should not, rely upon present tests. available at the K-1 levels... 
At these levels, the tests that do exist are more oriented to 

the concept of "readiness" than to that of achievement and 
v> require extensive 1:1 oral administration time. Thus, given 
limitations of using such- tests, the study would then focus 
on grades 2-4. 

t Since the basic p.oticn of an IndTV^dualized program i'S antTthetic^l 
to the use of grade level labels,' any grade level combinations 
used should be contiguous (i.e., grades 2 and 3 or grades 3 and 
4 rather than grades 2 and 4) in order to reflect the contiguous 
nature of the instructional development within individualized • 
programs. This constraint is desirable because slower students 
in the higher of the two levels might be using theMower level's 
materials and vice versa for faster students in the lower 
gr^de level. jMoreover,- content and^sequencin^ vary considerably 
, in individualized programs. 

♦ 

• The combination of grades 2 and 3 is preferred over the combina- 
tion of grades 3 and 4 because the former is feU to be more 
representative of the K-4 range overall. Additionally, it may 

be that instructional programs at' grade 4 are different ^ < 
enough in concept from those at grade 3 to make fhe 3-4 combina- 
tion less contiguous than the 2-3 combination. 

It is also suggested that for the purposes of analysis (see Task 5), 
all results for grades 2 and 3 should be combined, both for the individual- 
ized programs and for the standardized programs. Separate analyses by 
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grade level could always be generated if a coiflparison of results by a 
category such as the nominal grade level associated with a given program 
for- administrative purposes is considered desirable. 

for sampling purposes, a constraint of having to include about the 
s,ame number of nominally second grade classrooms and nominally third grade 
classrooms in each sampling cell would be necessary to reflect the grade 
level selection and discussion presented here. 

B. Instructional Setting 

The unit of analysis suggested for this study is the classroom. 
However, 'the special nature of the programs to be studied requires 
careful description of the "classroom" to be studied. The concept of 
individualization suggests that important differences in process and/or 
educational practice mdy exist dinotig different teacners within a given 
"indj/i dualized" program. Thus, many important process differences 
may be lost in an analysis which studies a unit larger than one tied to 
- the activities of a specific teacher. It is also Vecognized that differ- 
ences may exist among students served by the same teacher in, an 
"individualized" classroom in terms of process applied or service 
received. However,' since programs of general interest to the audience of 
this study are unlikely to have been implemented for only an individual 
student it. seems inappropriate to limit the unit of analysis to an indivi- 
_^ • dual student, however incongruous that position jnight seem in a study of 

individualized instruction. Individualization within a classroom is a process 
controlled by the teacher, and it is the interaction of the teacher with the 
students in that classroom that should be an objective of analysis. 
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Having stated the case for th^ classroom as the 'study's unit of. 

analysis, an additional complexity must be addressed^ Since thfe study is 

to' include programs of both the mainstream and the pull-out types, tijis 

issqe must be considered in determining the actual unit of analysis. This 

consideratitfn begins by understanding the basic nature of the comp ed 

readi,ng or math programs that form the overall universe for this study, 

a universe from which well -implemented individualized and well- 

' . . »^ . 

implemented standardized programs/are to be "drawn and compared on the 

ba$is of their ^effectiveness. ' The comp ed student receives the'benefit 

, of special funding provided by Congress, (or other legislative bodies at 

the state and/or local leve-1). 'This special funding is reflected in the 

comp ed school' by the presence of specially hired personnel .(e.g., 

teachers, sp^^al.ists, paraprofessionals) and/or specially purchased 

materials or equipment. 

The impatt of these special personnel and/or materials is felt 

through interactions of these staff and/of materials and the^comp ed 

students within these schools (not all students within comp ed schools 

are comp ed students) during instructional time periods. This instruc- 

tion might take place within a portion^ of theljrfegular classrroom with the 

special comp ed instructor working with the comp ed students while the 

regular teacher in that room works with the^non-comp ed students. Or it 

might take place within this same classroom with the regular teacher 

providing all instruction but with the assi,stance of supplementary 

materials for the comp ed students. Either of these operational 

structures would be considered a "mainstream" program. 
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Another means for conducting the comp ed instruction would be use of 
a classroom set aside especially as the comp ed learning center, Jn this 
case, the comp ed students would leave their regular classroom and receive 
•instruction from the comp ed personnel in the center. This program would 
be of the "pull-out" type. 

It is important to note that the mere existence of a center within 
a comp ed building does not label the comp ed program in that building 
as a pull-out type of program. The key .issue is whether the comp ed 
students receive their comp ed instruction while in the same setting as their 
non-comp ed classmates or whe-ther they ^e physically separated into 
different rooms for this instruction. If the non-comp ed students use 
both a base cJassroom plus the center for reading and math instruction, 
and the comp ed students in this class are always in the same room as their 
classmates even though the two groups receive different instructional 
programs in at least one of these settings, this program would be con- 
sidered a mainstream type. 

Further, v/hile the programs for this study will be classified and 
selected based upon the nature of the comp ed instructional activities, 
comp ed students may participate in non-comp ed instructional activities 
as well* as comp ed activities. It might occur, for instance, that a 
highly individualized comp ed program would operate within a 'Vegular" 
setting that is highly standardized. The importance of the interaction 
between the regular and comp ed programs may depend heavily upon whether 
the comp ed program is of the' main^ream or the pull-out variety. 
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It seems neither feasible nor desirable to build into the sampling 
strata all the possible interactions between the type of instruction in 
comp ed program and the type of .instruction in the regular program. All 
such interface considerations (e.g., total hours of regular plus comp ed 
instruction vs , the hours in the comp ed portion of this total) which 
cannot be built into the sampling design will be considered in the overall 
study analysis as best they can, though it is recognized that only those 
process differences reflected in the sampling plan can be investigated 
in a exhaustive manner, \ ^ ' ^ 

However, it is critical tcr the study that the instructional programs 
studied, v/hich will be initially classified on the basis of only the 
specific comp ed instructiorial activities, be defined for analysis purposes 

aV[ instruction provided to the comp ed student whether by or in the 
specif ic coiTip ed program or by the reguldr staff in Llie r eyular cldbSruum 
context. 

Thus, the "classroom" to be considered the unit of analysis is a 
special one, 'Exhibi t 'III-7 represents this "classroom" schematically for 
mainstream programs, This^ "classroom" -- 'indicated within the dashed 
boxes may indeed be one physical room (A) or may even be two rooms* (B) 
so long as the comp ed students are /lot physically separated from their 
classmates during comp ed instruction. The .program which we wish to 
investigate then would be the reading or-math activities of the comp ed 
students provided both by the comp ed program and by the regular teacher 
as part of the general program. 
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Exhibit III-7 



EXAMPLE STUDY "CLASSROOMS" FOR MAINSmAM^OGRAMS 



Sm)Y "CLASSROOM" A 



Room X 



Regular Instruction 
Corp Ed Instnx:tion 



STUDY "CIASS!m4" B 



Room Y 



I^ssming Con tor 







Regular Instruction 


Regular Instruction 






< ^ 

All Students 








Ccnp Ed Instruction 



ERIC 



i 
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Exhibit '111-8 represents the study classroom for pull-out program.. 
The program which we wish to investigate then would be the reading or 
math activities of the comp ed students served by both the comp ed ^ . 
program and by the regular teacher in the classroom from which the comp 
ed student has been pulled. The specific rooms to be encompassed in our 
study "classroom" in this case would depend on the specific students in 
the center whose results are to be sampled. ' 

C. Classifications of Programs 

As discussed in Task 1, each program studied will be classified 
"in a dichotomous (Yes/No) manner on each of the following four issues: 
§ "PO" Performance Objectives 
• "D/P" Continuous Diagnosis/Prescription 
9 "AlP" Fxif;tencp of Alternative Learning Patjis 
t "PACE" Individual Pacing 

For definitional purposes, the matrix shown in, Exhibit III-9 will form the 
basis "for program classification. The cells of this matrix have been 
numbered from 1 through 16 as shown. The pattern of Yes/No responses to 
the four preceeding issues are also shown within each cell with the 
responses associated with that cell listed from top to bottom in the 
ceil corresponding to PO on top, D/P next;, ALP next, and PACE on the ^ 
bottom. 

Note that six of the 15 cells have been shaded. These are cells 
which contain two "Yes" plus two "No" responses. These shaded cells have 
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Exhibit in-8 



EXAMPLE STUDY "CLASSROOM" FOR PULL-OUT PROGRAMS 



Roori Z 



Regular Instruction 



Learning Center 



Only the 



'CoTp Ed Students 



Comp Ed Instruction 



ERIC 
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Exhibit II I -9 



DEFINITIONAL MATRIX 
V 





PO - YES 


PO - NO 


D/P - YES 


D/? - NO 


D/P ~ YES 


D/P - NO 


W 
1 

•J 
< 


PACE- 
YES 


^ Y 
Y 
Y 
Y 


2 Y 
N 
Y 
Y 


3 N 
Y 
Y 
Y 


Y 
Y 


1 

w 

o o 
< ^ 


5 Y 

Y 
Y 

N 


] 

N Y 
Y| . Y 

Yi \ K 


« N 
N 
Y 
N 


ALP - NO 


PACE- 
YES 


3 Y 
Y 
N 
Y 


10 

N 
Y 


Y 
N 
Y 


N 
N 

- N 

^ \ V 


1 

o o 
< is 


y 


N 
N 
N 


N 
Y 
N 
N 


N 
N 

N 



ERIC 
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been excluded from, the anticipated analysis s-ince no clear pattern of 
individualization or standardization emerges. This is the case because 
the study design team chose to establish no hierarchy of importance among 
the four key .issues reflected in the matrix. It should be clear that the 
design reflected in the overall matrix is factorial in nature. Indeed, 
if programs for all 16 cells could be located in a reasonably balanced 
manner, the factorial design reflected would allow a thorough evaluation 
of each of the four key issues (main effects) plus all the relevant inter- 
action among these four key issues. If this were true, the analysis would 
be a study of these issues, and programs would not require further 
labeling. However, the study design team considered each of the cells 
in the definitional matrix with great cire -- reflecting the wide program , 
experience available within and to the team — and determined that 
logically not all of the 16 cells 3?e equally likely to be found (i.e., 
certain specific patterns of Yes/No responses to the four" key issues would 
not exist too widely in practice). Thus , the full factorial power of the 
matrix was not to be available. Given this prac'hcal limitation, it was 
decided to esta^)lish the four study groups described below. 

§ Group I all "Yes" responses; considered to be /he most 
individualized programs given the structured 
t definition of an individualized program as 
suggested here; includes only cell number 1. 

t Group II three "Yes" responses plus one "Mo" response;- 
considered to be the, next most individualized 
group of programs within the suggested definition; 
includes cells numbered 2, 3,' 5, and 9. 
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Group III one "Yes" response plus three "No" responses;^ 

considered to be almost 'totally standardized (or 
traditional or non-individual ized) programs 
within the suggested definition; includes celTs 
numbered 8, 12, 14, and 15. , ^ 

% 'Group IV all "No" responses; considered to be the most- ^ ^ • 
standardized programs given the suggested definition 
includes only cell number 16* 

The four groups may be summarized" as follows: * ' 

very individualized programs 

individualized programs 

standardized programs ' ' *' 

very standardized programs * 

The reader is cautioned from viewing this definitional , matrix as ^ 
one which somehow maps the distribution of actual comp ed practice. 
Such a view might lead one to conclude that excluding the sJhaded" portion * 
of Exhibit III-9 from the analysis would exclude aOarge n.umber of " pro- 
grams from potential consideration. .The study team does not believe this 
to be the case based on previous experience as well as specific exper- 
iences gained in identifying candidate sites during this design phase. 
However, even if it were the case, the RFP expressly states that NIE, iS 
not interested in a study of rep»*es6ntative individualized practice in 
comp ed in the U.S. today. Such a surv^ — useful for broad policy 
decisions — is the subject of District Survey I. This study at hand is 
tasked to contrast well-implemented. individualized programs and well- 
implemented' standardized programs and the sample selected for study need 
not be reflective of the average experience in this regard in the U.S. 
today. , ' 



• Group I 

• Group T I 

• Group III 

• Group IV 
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However, if the study implementation contrasts were to discover a 

significant number of potential lyjj^eresting programs falling into aU 

16 cells of Exhibit III-9, the. reliance on the above defined four study 

groups could be reduced in favor of additional reliance on the factorial 

power' of the basic matrix. The contrasts to be addressed would shift 

from comparisons of effectiveness between or among groups to a considera- 

ti^ of the main effects and interaction effects contained with the 

matrix obtained. The TURNKEY study design team does not feel that this 

is a likely event; it is mentioned here only in the interest of completeness 

• * 

Exhibit III-IO sumrnarizes the design framework of the overall 
sampling structure, " . " 

Sampli'ng C riteria 

♦ 

In addition to the framework into which all selected programs must 
fit, a number of specific criteria must be met by such programs. These 
criteria relate to the following considerations: 

• Practicality the necessity for selecting programs for 
which' data collection may be conducted in an efficient 
manner with LEA cooperation ensured; 

• Design — the requirement for programs which meet the 
design specifications outlined by NIE and/or included in 
the proposed design; 

• Stability ,the need for selecting programs which reflect 
a settled program which is operating in a stable mode and 
has overcome start-up and disruptive problems; 

• Implementation the requirement for available programs 

of sufficient scepe that meaningful implementation data can ^ 
be collected and verified prior to final selection. 
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EXHIBIT III-IO 



SPECIFIC DESIGN FRAMEWORK 



GRADES 

NOMINAL COMP ED mSTRUCTIONAL SETTING; 
SUBJECT AREA: 
UNIT OF ANALYSIS: 
PROGRAMS STUDIED: 



2 and 3 Combined 

Mainst^^ 

Reading 

"Classroom" 

Groups I , II , III , Iv 




GRADES : 

NOMINAL COMP ED INSTRUCTIOlJAL SETTING: 
SUBJECT AREA: 
UNIT OF ANALYSIS: 
PROGRAMS STUDIED: 



2 and 3 Combined 

Pull-Out * 

Reading" 

"Classroom" 

Groups IV .II, III, IV 



GRADES: , 

NOMINAL COMP ED INS 
SUBJECT AREA: 
UNIT OF ANALYSIS: 
PROGRAMS STUDIED: 




SETTING: 



2. and 3 Combined 

Mainstream 

■Math , ■ . 

"Classroom" 

Groups I, II, III, IV 



GRADES: ^ ^ ' 

NOMINAL COHP ED .INSTRUCTIONAL SETTIN.6: 

SUBJECT AREA: 

UNIT OF ANALYSIS: ■ 

PROGRAMS STUDIED: 



2 a:n3 "3'^ Combined 

Rum -Out 

Math 

"Classroom" 

Gfcaup^T>,II,, III, -fV. 
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Exhibit III-ll describes the specific sample selection criteria which 
'will be followed in the sampling process. It, also specifies the general 
rationale(s) for tfTe use of each criterion in the selection and identifies 
the relevant items in the two '(district and school level) sampling quesj:i6n-^ 
aires which will be discussed later in this section.- 

Sample Selection Procedures 

The procedures by which the final sample will be selected are dictated by 
the nature of the universe(s) from which ^the districts and schools are to 
be chosen, ' ' . - 

Accordijig to NIE specifications for the study, the final sample is 
to contain programs selected from those included in NIE's District Survey 

as well as those chosen on the basis of the investigation during this 
study. In our interim progress report, submitted to rUt' on 1 August 1975, 
we recommended that this specification-be el iminated and that the sample . 
selection -procedures fior the two studies be completely divorced'. In the 
RFP arid from 'conversations with NIE s^taff , it t^s been stated that a major 
purpose o^.thfs sample "ovei^lap" between the two studies is to provide 
validation (in the form^of classroom observations) of the data gathered in 
.questipnnaire form, during District Surve^Hj^^from^t^^ and admin^i^s-^ 
trators. Although' spot validation gf these ^I'uestionoaire responses is 
•clearly necessary, the advisability of using (and possibly confounding)^ 
the large-scale Individualized Instruction Study as a'Checking mechanism is 
open to question. It appears more scientifically sound for a validation ^ 
mechanism to be bui.lt directly in^the design of District Survey I or -to 
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Exhibit ..1 1 1 -11 
Page 1 of 2 



CRITEJRION 



QUESTIONNAIRE 
ITEM (6) 
(D-DISTRICT) 
(S-SCHOOL) 



RATIONALE 



M 



M 



(J H 

H D 

CO Q 

Q S 



3 

Eh S < 
CO HQ 



Willingness to participate rnust be 
indicated at both the IFA and the school 
level. 

^.^^-school must either b^ a Title I school 
or a Title I eligible school. 

A program must have been in operation in 
the LEA^r at le^st th4 1974-75 and the. 
1975-7€^^school years, and in the school 
since at least the 1975-76 school year. 

, • * 
A "proyxdiu iCiuSL have fcocn ccntiriUGv^ frcrn 
the 1975-76 school year to the 1976-77 
sclxx>l year in the study claSgroorns with- 
out ma jor changes in operation. 

A prograti raust be in operation in at least 
10 grade tv.o classrooms and at least 10 
grade -three classrooms in the ip\ and 
at both grade levels in the study schoQls. 

A classroom must contain a projection of 
at least 6 oanp ed students in the program 
of interest for mainstream programs or at 
V 6 ccmp ed students from any regular 
classroom served by the center for pull- 
out programs. 

' ■ • . ' 

^to recently irrolemented "najor" student, 
redistribution* that has affected more 
than 15% of tha enrollment in potential 
study schools (e.g., no busing orders or 
plans, no redistribution of students due 
•to clsoing of other schools with 
declining enrollments) since 1 Septpiiber 
1974. - ■ ■ 



D-13 
S-19 



S-7 



I>-11(7), 
12(7) 
S-14(7) 



D-ll!^ 9) 
12U, 9] 

S-14(S, 9) 
.15(J, 2, -f] 



D-11 
12 
S-14 



(2, 
(2, 
(2, 



I>-7, 
S-8, 
S-15 



8 
9 

[5) 



S-16 



8-1' 

1-35 
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X X 



X X 
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RATIONALE 


CRITERION^ 


QUESTIONNAIRE ■ ~ 

ITEM(S) 
. (D-DISTRICT).» 
. (S -SCHOOL)' 


PRACTICALITY 


DF.SIGN- 
REQUIRED ■ 


STABILITY 


§ 

H 

1 


No classroon will be included in the 

SL.LlQy WilwOvi Uticiv^ 1 Itil. Xo IICW uv-' L-i ic. ^i.^^^'v^JLcajii 

during the 1976-77 school year; nore , 
specifically, for minstream prograins, 
no classroom v;hose conp ed teacher is new 
to that classrocm that year will be 
included and for pull-out prograir^s none 
or tme comp oa lc^cu^^s i>cixv_uiy yLautjo 
2-3 carp ed students can be new to the 
proj^am that year. ' , ' 


S-15(4) 






X 


X 


A school must have a non-inyiority 
population percentage which fails withiH 
1d% oz: tne oistrict s overall iiuie i 
school/ non-minority population 
percentage • 


S-13 


X 


X 






A procji.«rn niusu nave aueuuciLu uc'v^uiiit-iiua.uxtw'ii 

in the form of curriculum' guides, 

suggested teacher lesson guides, or''som2- 

thing similar and district or school- ' 

le\^el nandates regarding these guidelines. 
» 


D-1 1 114) 
12(14)- 
S-14(/4) 


X 




ft 


X 


✓ 

At least .80% of the study body in a school 
must' speak English as their natiVe 
language. 


S-12 


X 


■ X 






No more than one generic type of comp ed 
program in reading and one in irath per 
school building iray be available^ in a 
study building. 


S-14(/) . 






. X 


X 


Both the district level and school level 
program administration must have checked 
at least the next to the highest level 
ipplen^ptation/effectivencss response iit 
their respective survey questionnaires. 


I>11(6) 
12(6) 
S-14{6) 


' i 






X 


r!c ^ 11 
1 
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commission a small validation survey as a separate entity-. For a number 



of reasons, the requirement that 50% of the sample for the Individualized 
Instruction study be selected from the District Sut?vey I sample may result 
in confounding of the results of both studies: 

1) Because District Survey I data will be collected 
during the 1975-75 school year and the individualized 
instruction study data will be collected during the 
1976-77 school year, it will be difficult to determine 
if apparent inconsistencies are really discrepancies 

or merely reflective of changing program characteristics 
(Note: our research indicates many individualized 
.programs are undergoing continuous structural change). 

2) ' Should validation by observation reveal apparent 

weaknesses in the District Survey I questionnaire 
procedure, no effective remediation process is 
available. 

3) In the event that apparent discrepancies are un- 
covered, it is possible that unproductive disagreement 
between the two contractors regarding the efficacy of 
both Questionnaire and validation techniques cduld occur. 

4) It is possible that the District Survey I sample does 

not include enough sites which are suitable for inclusion in 
the Individualized Instruction study, in terms of program 
stability, si.ze, or willingness to participate. 

5) The validation role of the Individualized Instrucjtion 
study may reqjjire that its data collection instruments 
be structured in a manner which is not in the best 
interests-.of ' the ovfiraU analysis plan. 

6) Data availability from District Survey I may cause 
severe time .constraints to be placed on the sam{i]e 
selectit)n activities necessary for the Individualized 
Instruction study. 

For these reasons, which were indicated in our interim report, we recommend 

that this sampling restriction be lifted from. the study design. We 

hasten to note that this clwnge would in no. way preclude a District Survey I 
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site-frora inclusion in the Individualized Instruction study, should it 
meet the appropriate selection criteria for both studies. " 

Whether or naWthe sampling ties between the two projects are severed, 
the sample selection procedures will remain essentially unchanged/ tinder 
both sampling designs, a preliminary data collection activity will be 
necessary, during the early part of project implementation, before final 
selection of programs can actually occur. For all programs included in 
District Survey I, relevant items will be included in the teacjier 
questionnaire and possibly elsewhere for that study. For other potential 
candidate programs, preliminary questionnaires requesting the necessary 
information for the final selection of programs will be sent to appropriate 
school districts. Appendix 5 contains these suggested prel imi nary sanipl ing 
questionnaires and their cover letter. Based upon the responses to these 

4..:/ .^^^^ ^-^^U ^^^.'^r^-*--; -% 1 ov^Ar»v*:im iwi 1 1 Ko accoccorl y^ola^iv/o ^r. fha* 

sample selection criteria described earlier in this section. The 
thirteen selection criteria .wi 1 1 be applied in the indicated priority order 
(as shown in Exhibits III-zH and III-14) to each potential program until 
the appropriate number of ^ograms (including the required 25% oversampl ing) 
have been chosen to fill the appropriate cells of the .sampling matrix. ' 
In addition, this preliminary sample will include an additional 25% 
oversampl e of programs, which will serve as -replacements for selected 
programs found unsatisfactory upon on-site verification. 

■ fxhibit III-12 depicts, in flow chart form, the Sequence of samp-le 
.selection activities. In this paragraph, we expand.1>pon the activities- 
noted in Exhibit III-12:. . ~ 
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SA\1PLE SELECTION PROCESS 



ExJiibit II I -12 
Page 1 of 2 



DISTRICT SmXTi I DATA ' 

OBTAIN DISTRICT 
SURVEY I RESPONSES 



QUESTIOXN.AIRE DATA 



ASSBBLE DATA ON 
POTENTIAL PROGRAMS 



SORT DATA BY LEA 



T 



DISTRIBLTTE DISTRICT 
&ANPLIXG QUESTIOXNAIRl-l 



OBTAIN DISTRICT 
QUESTIONNAIRE DATA 



SORl" .-UL DATA BY LEA i 




APPLY SELECTION U 
CRITERIA AT DISTRICT ^ 
LEVTZL 1 






* 

QovrAcr PC 

LEAs ■ . 


)TE\T1 \L p 
* \i 

' 9 



/ 



CONU'JCr-" 15RIEF DATA 
\'ERIITCATIOX SITE 
VISITS 
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Exhibit III-12 
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APPLY SELECTION. 
CRITERIA AT SQIOOL 
LEV'EL 




ESTABLISH POOL OF 
ACCEPLABLE LEAs/ 
Sa^00LS/CLASSR0O}lS 



R^VDOMLY SELECT 
CLASSROOMS TO FILL 
MATRIX 



IDFoNTin' STUDY 
CLASSROOMS 



ERIC 
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Exhibit III-13 

. • V- 

SELECTION CRITERIA APPLIED AT DISTRICT LR'EL 



ERIC 



WILLING TO PARTICIRATE? 



Yes 



PROGR.-\^•I^il-\S BEEN IN 
^XISTFA'CE SINCE AT tE\ST 
1974-75 SaiOOL YE-\R? 



Yes 



PR0GR.V^1 IS TO BE CONTINUED 
TO 1976-77 SOIOOL YL\R 
PvELVniELY L'>.'QLV\GED? 



Yes 



PROGR.VNI OPERATES IN AT LL\ST 
10 GRVDE /nVO AND 10 GRADE 
THREE CL-ASRDONB? 



Yes 



DOES DOCUMI-XTATION EXIST 
FOR TFE PROGRANl? 



Yes ■ 



ILAS AT LE.-\ST IW. NEXl'- TCTlfl- 
MIQ'IEST LE\T-;L IMPLEMENTATION/ 
EFFl:CTlVT:Ni-SS RESPONSE BEEN 

an-:cKi-;D for im PROGRAii 



Continue ' 
*An LEA may "have more tliafi one program 



No 



No 



No 



No 



No 



No 
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ELIMI^JATE LEA 



ELIMINATE 
PROGR/VNI* 



ELIMIN.ATE 
PROGRAM 



elimin.\te 
progra:^! 



ELIMIN-ATE 
PROGRAM 



. -LIMI^JATE 
PROGR'XM 
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Exhibit III-14 
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SELECTION' CRITERIA .APPLIED AT SQIOOL UNEL 



WILLING TO PARTICIPATE? 



Yes 



SaiOOL DISTRICT TITLE I 
OR TITLE I ELIGIBLE? 



Yes 



PROGIl-V-1 EXISTED SINCE AT . 
LEAST 1975-76 SQiCDL YL\R 
;N •SQIOOL? 



Yes 



PROGRAM TO BE CONTIXIT^D IN 
1976-77 SQIOOL VlilR 
RELATRnLY irNQLANGED? 



Yds 



CLASSROOMS I\^ PROGR.V^I A\v\IL- 
ABLE WITH AT WAST SIX 
CO>P ED STUDILNTS? 



Yes 



NO f>DRE^ TfLAN IS'i Og SCUOOL'S 
ENTvOLLNENT AFHrrED BY MAJOR 
STUDENT REDiSTRlBUTlOJ." SINCE 
1 SEFrF::-BrR 1974? 



No 



No 



No 



No 



No 



ELIMIN.ATE 
SaiOOL 



ELIMIN-AfE 
'SaiOOL 



ELIMIXATE 
SaiOOL 



ELIMLWTE 
SCHOOL 



ELIMINATE 
SQIOOL 



ELIMINATE 
SaiOOL 



Exhibit III-14 
Page 2 of 2 



ERIC 



[ 



ARE TE-\aiERS IN ^TOTE CUSS- 
ROOMS TO BE ]/l\A'OL\'ED in/ 
PROGRAM NEXT YL.m 



Yos 



DOES SaiOOL 1-.\LL Kl'nilX 15^0 
OF DISTRICT'S 0\IIR\LL TITI-E I 
SCHOOL NOX-MlXORm' POPULA- 



Yos 



DOES DOOr-li'XTATION EXIST 
FOR PROGRAM? 



Yes 



AT LFAST 80°6 OF STliDEXT BODY 
SPE\KS EXGLISII AS TFEIR 
XWIVi: LVXGJAGE? 




Yos 



NO >DRE TIIAX OXE CO^fi' ED 

miADixG AXD oxT, caip r;D mv 

PROGRVI IN "SQiOOL? 



Yes 



lli\S AF LEAST TIE XT;XT TO 'H-IE 
IIIQIEST LLAT3L IMI^LD&IXTATIOX/ 
EFFECTIVENESS PvESTOXSE BliENi 

aiECiciiD FOR rj\ai pro(>r\m? 



Continue 
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No 



X"o 



No 



Xo 



No 



No 
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y 



ELIMIN.\TE 
SCi^OOL 



ELIMINATE 
SCHOOL 



ELIMINAIE 
SQiOOL 



ELIMIN\TE 
SCHOOL 



EEIMIN\TE 
SaiOOL 



ELIMINATE 
SCHOOL 



\ 



Obtain District Survey I Responses For all sites included 

in District Survey I, appropriate data will be obtained from SRI; 

Sort Data by LEA This District Survey I data will be sorted by 
LEA; 

Assemble Data on Potential Programs — Based upon data accumulated 
through literature searches, printed matter, and materials ^ 
provided by publishers, agencies, etc., all data on potential 
programs will be assembled; . , ^ ^ 

Distribute District Sample Questionnaire (see Appendix 5) 
The preliminary district-level sampl ing ^questionnaire 
will be sent to the superintendents of the appropriate LEAs; 

Obtain District Questionnaire Data — Responses from the 
District sampling questionnaire will be received; 

Sort An Data by LEA — All program data will be sorted by . 



Apply Selection Criteria^^at District Level The thirteen samplj§^ 
selection criteria will be applied^ where appropriate, to the 
data on LEAs. LEAs which do not meet these criteria will be 
el1i!iinated. Potential schools within each acceptable LEA will 
be identified. Exhibit IIN13 illustrates this procedure in 
f I'^th?^ deta ^ > 

Contact Selected LEAs — The LEAs which are selected 
according to the selection criteria will be contacted 
to arrange for site visits; 

Conduct Data Verification Site Visits — Each in which a^ • 
selected program is located will be visited by a member 
of the project team; 

Apply Selection Criteria at School Level Appropriate 
selection criteria will be applied to the school-level data 
gathered during the site visit. Exhibit III-14 illustrates 
this procedure in further detail; 

Establish Pool. of Acceptable LEAs/Schools/Classrooms — A ^ 
pool v/ill be established of all classrooms (by LEA and, 
school) which meet. the sample selection criteria; 



Randomly Select Classrooms to Fill Matrix — * From the pooT of 
acceptable classrooms, individual classrooms will be randomly 
selected, one at a time, until each /Orking cell of the 
sampling framework is oversampled by 25%. 



LEA; 




t . Identify Study Classrooms From those classrooms 1n each 
cell, those which v/ill be considered as primary study class- 
rooms will be identified. 



Final Sample Dimensions 

Appendix 6 descrjibes in detail the statistical process followed for 

determining the saj^fple sizes presented in this portion 'of our report. To 

summ^Hze this appendix-, the'following aVe the key assumptions that are 

reflected in the numbers presented here: 

t The most demanding (from ar statistical point of- view) comparison 
that the study will be forced to ma ke~' would involve a univariate 

* t test between two independent groups selected frorji the 36 

* cells defined in Exhibit III-IQ; a significance leyel of .05 
with a 2-tailed test has been assumed. 

• Group' means from our "classroom" units of analysis will be > 
utilized. These means will be based on approximately 6 
students for mainstream program^ cind, approximately 10 
students from pull-out programs. . ' ^ 

e The effect size which' is considered desirable to detect, -i.e., 
educationally significant, is 0.5 standard deviation difference 
between two population means. 

0 A conservative estimate of the intraclass correlation of 

achievGinent scores (i.e., correlation of scores from students 
within the same classroom rega^rdless of treatment) is 0.5. ' 

Q The level of statistical power ('i.e., the probability of 

detecting a true difference of 0.5 standard deviation in two 
population means) used here is .90. 

Based on these assumptions and the more detailed discussion contained 
in Appendix 6, Exhibit 111-15 displays the suggested number of classrooms 
required for each of the 16 study cells implied in Exhibit III-IO. The 
total number of classrooms included in this display is 800, vrTth no allowance 
for oversampling. Usin^. a 25% oversampling rate, a total of 1,000 class- ' 
rooms would be included in our proposed study design. 
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EXHIBIT 111-15 



NUMBER OF- CLASSROOMS NEEDED IN THE STUDY 
SAMPLE, WITH NO 0VERSAMPLIN6* 



PROGRAM TYPE 







GROUP I 


G"R0UP II 


GROUP III 


GROUP 


A. 


Mainstream Reading • 


51 


- 51' 


5V ■ 


51 


B. 


Pul.l-Out Reading 


' 49. : 


49 


49 


49 


C. 


Mainstream Math 


51 


51 


. 51 . 


'• 51 


D. 


Pull-Out Math- ■ 


49 


49 


49 


49 



*UsingL a 25% oversampling factor, a total of 1 .000 classrooms-are" needed". 
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• Recalling that study Groups II and III each include four cells from 
witjiin. our definitional matrix while study Groups I and IV each include 
only one cell for this matrix, it is apparent that not each individual 
cell in our definitional matrix need be equally represented to^'fuTfTl^ 
the sample dimensiojis listed here. However, to the extent possible, the 
individual cells within each of pe multi-celled study.group should be 
represented in a relatively balanced manner to facilitate any fpntrasts 
to be made which call for data to be disaggregated to. that level of detail 
(see Task 5, Level 2 Analyses). For the overall study design constraint, 
hov/ever, it is only important that the numbers listed tn Exhibit 1 11-15 
be adhered to, if the sample size determiniation process used here is'. to 
be relevant. 

The earlier references to proportional sampling should be considered 

.1 • * 

at this time. If the implementation contrast should find thdi for insLdnce 
overall only 30^. of^ the programs available for this study are of the pull- 
out variety or that only ^0% of the available programs can be classified 
•as Group IV (neither of these illustrations are felt to be accurate, .however) , 
the nymber of required sample classrooms reflecting these dimensions could 
then be adjusted to reflect these reality limits. ^ This adjustment could 
be made by maintaining all those feasible or readily attainable treatment 
categories^ at the level shown in Exhibit III-15 while adjusting downward 
the sample requirements not-s6-readily attainable treatment categories such 
that the ratio of the non-adjusted^ sample sizes to the adjusted ones 
reflects the^ ratio the numbers of such programs overall. lo the first 
of the hypothetical adjustments indicated by our illustration, the second 
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and fourth lines of Exhibit III-15 would be reduced from 49 to 21 to reflect 
the 70/30 ratio indicated in program availability data. In the second 
such adjustment, the fourth column of Exhibit III-15 wouVd be changed from 
51, 49, 51, 49 to read 17, 16, 17, 16 to reflect the 30/10 ratio 'between 

r 

each of the other three columns and the fourth column indicated in the 
program availability data.. 

No need for reliance on proportional sampling as described above has 
been identified to date. The difference in sample siz^ibetween the main- 
stream and pull-out programs reflected in Exhibit III-15 is solely based 
on the differentia-l assumptions used for the number of students from each 
type of classroom setting to be tested for purposes of this study, not on 
any assumption or data on the basic avciilabil i ty of programs from either 
of these treatment strata. 

Additionally, it should be recallec| that the study design calls for 
approximately the same number of ^minally second grade classrooms as 
nominally third grade classrooms reflected in each treatment category. 
This means that a))Out 24-26 classrooms at e^ach of these grade levels would 
be includedj'n each program grouping. 

As a closing note on the suggested sample dimensions, it may be 
estimated that the 1,000 classrooms discussed in this section will reflect 
the following numbers of programs and LEAs: 

• X Assuming 10 classrooms per program on the average, a total 

of approximately 100 programs will be studied. 

• . Assuming 4 programs per LEA, on the average, a total of 

approximately 25 LEAs will be involved in the study. 
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., The nature of'the sampling plan, especially its reflection of our 
definitional matrix, allt)ws a number of issues critical to NIE and the 
prime audience of the proposed study (i.e., Congress) to be addressed in 
a highly structured fjianner. Though a true experiment (wherein all else 
is kept constant while only one program feature is systematically varied 
at any given tinfe and students are assigned to treatments randomly)! 
is j'mpossible- to achieve in any real world setting, the quasi-factorial ^ 
structure suggested here allows an approximation of at least a portion of 
this scheme to obtain. (The reason for not ful«^y utilizing the factorial- 
interact potential of our definitional matrix was stated earlier.) 
Careful selection of programs and matching of students in adherence to 
the sampling contrasts listed earlier wiTl suffice as the suggested approx- 
imation of random. assignment of students, recognizing that this approximation 
ha^ been a major problem for all large scale field research and evaluation 
studies in education. 
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TASK 3 - DEVELOP PROCEDURES FOR ASSESSING PROGRAM IMPLEMENTATION 



^ Summary of Critical Issues 

In our Proposal, we identified^(and discussed rather extensively) a 

• * 

number of critical issues regarding the assessment of program implementa- 
tion. In addition, we discussed several proposed approaches to resolving 
these issues during the design phase. 

First, the critical issues identified and discussed in our Proposal 
included: 

• The need conceptually to separate well-imple(nented programs, 
from highly-individualized ones (i.e,, a standardized progi?am 
could be well-implemented) from highly effective programs, 
recognizing that interaction exists between these measures 
(i.e., a well-implemented plan may not be a highly i ndi vidual ize'd 

, program, and conversely, a highly-individualized program may 
not be wel 1 -implemented. • 

• The difficulty inherent in obtaining/developing a baseline 
regarding the nK)st effective and valid prescribed plar> for 
implementation, in light of contextual restraints* (e,g. , 
LEA budget .constraints precluded tne purchase of all supple- 
mental n^aterials designedifor use in an individualized reading 
center), the intentions of publ.ishers (e.g,, the desire to 
accommodate teacher desires as a trade-off against effective ^ ^ 
implementation), and other factors. 

• The need to resolve an apparent conflict between the inclusion 
of only well-implemented programs in the study and then 

\ assessing the impact of varying degrees of implementation on 
program outcomes (see RFP). 



In addition, during the design phase, we identified several additional 
critical issues, including: a) whether or not a single uniform instru- 
ment could be used to determine in a, valid manner the degree* to which 
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specific individualized programs are well-implemented; and b) the degree * 
to which cross-()rogram comparisons relating degree of implementation 
to outcomes could be ma^cje. 

Second, implementation, in the context of the RFP, was taken to 
mean some combination of stability over time (i.e., steady sta]^e operations 
beyond the transiency of any start-up activities), plus operation in 
accordance with the plan as initially set out or as modified' prior to 
the school year under study, 1975-77. This view of implementation lends 
itself to a two-step process for ervsuring that only those programs that 
are likely to have been well-implemented are selected for*study. Sub- 
sequent verification through on-site observations and other data collec- 
tion procedures will determine whether the programs studied are indeed 
well-implemented, and if not, the reasons v/liy. 

As described in the Proposal, a nuti^ber of sub-steps were discussed 

> 

identification, including review of existing program implementation 
gu-ides, results of studies with similar purposes (e.g., the SRI 
Follow Through study), and initial discussions with LEA staff members 
involved in candidate projects. A great deal of, time was expended in 
attempting to resolve issues surrounding the second step (i.e., determining 
the degree to which* potentially well -implemented programs are indeed 

well-implemented) . In the following pages we discusjuJtlie--ac^^ ^^""'^ 

undertaken^dyring this task, the relevant findings and results of our 
efforts, and the procedures used in developing instruments used during 
the design phase and thoserfor developing additional instruments to be 
used during the implementation phase of the study'. 
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* Approach Taken/Findings 
.During the design phase, procedures a^ndHTiH!;^^ 
were developed to identify potential candidates and determine tentatively 
whether or not candidate programs w^re ^el 1 -imple/iiented. Instruments 
were also developed to verify whether or not potential candidates met 
minimal criteria for inclusion in the study (see Appendix 5), Lastly, 
instruments and procedures must be fully developed and field tested 
for collecting data (through observation, surveys, and audits) on the 
implementation of programs during the school year, 1976-77. Under this 
task we discuss the procedures and instruments which have been developed 
and used thus far in the design phase as well as the specifications 
and procedures for developing additional instruments during the 
implementation phase.^yft should be noted that the overlap between 
this task and the, previous task, sarrple selection, and with the site 
selection procedu^'e desc^^bed in Appenriiy 7 is pxtpnsivp: to eliminate 
redundancy, appropriate references have been made for the reader. 

Identification of Well-Implemented Candidate Programs < 
As' described in Appendix 7, upon design contract award the TURNKEY 
project staff initiated several procedures to identify potential can- 
didates whose programs, individualized or standardized, were we'll- 
implemented. Since obtaining 0M3 clearance for a formal survey during 
the design phase was considered infeasible, existing documer^ation and 
knowfed.ge had ta be heavily relied upon. Minimal criteria f6r inclusion 
included: a) that the program covered grades K-4, math and/or reading, 
and focused on compensa^ry , education target groups; b) that documentation 
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was readily accessible in the fonn*of program descriptions, evaluation 
reports, validation findings, paneTreviews , or Personal observations of 
the project team members. A list of potential candidates (see Appendix 
8) was gathered from: Right To Read demonstration program (USOE); 
USOE Title I and Title III offices, including programs which were or 
are presently being reviewed by the Dissemination Review PaneJ 
of USOE; sites and .potential sites for inclusion in the Program Informa- 
tion Package program (USOE^); and the TURfJKEY documentation file of al- 
most 100 projects in 20 states. In addition, the major publishers 
o,f individualized learning systems were contacted and their nomination 
of the five districts which were implementing their program in the most 
effective manner was solicited. 

In order to determine whether potential candidates were "likely" 
candidates, meeting minimal criteria, checklists were developed. Tha 
first, enclospd in Appendix 9, was developed in August and submitted 
to NIE for inclusion in District Survey I as an instrument to assist 
in sample selection. This instrument was similar in most respects 
to' a second instrument developed by TURNKEY staff and used to determine 
whether potential (non-Survey I) sites met m-inimal criteria and thus 
qualified as likely candidates. This Instrument went through two re- ^ 
visions and was revievyed by three LEA consultants and other education ^ 
officials, to determine its practicality and feasibility in light *of 
the nature of generally existing documentation. This checklist is 
enclosed in. Appendix 10. ^ 

"Next, the checklist was applied to potential candidates by 
members of the project * team. .*In all instances, at 'least half of the 
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information needed to complete the checklist had to exist j'n documen- 
tation available to project team members. Upon partial or full comple- 
tion of the checklist, project team members talked with individuals • 
responsible for or knowledgeable about the respective programs at the 
■ respective sites to complete the checklist and/or t&^erify certain 
portions of it. Upon completion oi^ the checklist procedures the likely 
candidates were assigned to the respective sample cells, described in 
Task 2. In addition, the information enclosed ir> the identification 
checklist (Appendix 10) was punched on individual Keysort cards as 
likely candidates for future consideration.' 

We are aware of some of the limitations of the above procedures 
for identifying well -implemented programs.^ However, in light of the time 
and other constraints mentioned earlier, this procedure was felt to be" 
the most feasible. First, we assumed that the publishers of programs 

iomont;^tinn ouidpc; and/oT checklist 
procedures have designed them to insure greater program effectiveness 

« 

and/or acceptance by teaching staff. While we feel comf ortabl e -wi th the 
sites nominated by the publishers, we are also aware of the need to 
review implementation plans to ensure that "-suggested" procedures do 
increase the probability o'f\ high student performance rather than • 
increased sales. 

•Second, Titl6 I and Title III projects which have'been validated 
and, -particularly, those which have been approved by the Di,ssemina,tion 
Review Panel, have received rather extensive evaluations oyer at least 
two years and in most instances have been validated not only by federal- 
officials but a'lso by state education officials. A review of validation 
forms and procedures used by USOE and several SEAs indicates a strict 

103 

■ - III-54 



adherence to documented plans and objectives and the degree ta which 
the programs followed plans and met objectives to be a relatively good 
baseline for assessing implementation. As described later, however, 
it is important to note that verification through survey and on-site 
observation prip\: to final selection will^be necessary, as described in ^ 
Task 2 and below, \ ^ 

In an attempt to-develop procedures and instruments for assessing 
the degree of program implementation during the s^udy implementation phase, 
we under^ok the following tasks: a) ,a review of "critics" of approxi- 
mately 20 cominonly-used math and reading systems which could be or were 
predominately implemented in ah individualized mode (ERIE Reports 46 and 
65«); b) a preliminary analysis of tKfe implementation guides/manuals 
provided by publishers, regarding specific learning systems; c) an 
andly?;is of observation and ot^her checklists used by publishers' con- 

9sultants and implementation checklists developed by external groups 
(e.g., /I/D/E/A/ Ketteri ng. regarding the IGE program); and d) a review 
of recent studies which have attempted to identify implementation variables 
and, assess the degree to which implenientation differed among the various 
programs, particularly the SRI Follow Through study. 

In addition, proj^ci members had an opportunity to discuss 'the 
proble^ms .and approaches in assessing implementation with: a) three represen 
tatives from /^AD/E/A/who shared their time, internal checklists, and, 

"^e unpublished results of their study, utilizing these checklists with 
the project members; b) Federal officials and SRI staff involved in 
the Follow Through study; c) numerous Follow Through directors whose 
model s -were assessed on the SRI implementation scale; d) Federal and other 
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consultants who provided technical assistance during the' implementation 

* 

phases of the Right To Read program; ^) LEA consultants; including 



principals and other staff, involved in the implementation of Title I 
programs; and f) several con^sul tants Vesponsible for trai ni ng, Title I 
teachers and other staff in the implementation of a variety of in- 
dividualized learning systems. 



Based upon these activities and reviews, the advantages and disad- 
vantages of two approaches for assessing program implementation during 
the operational' phase of the project were considered: a) whether or not 
it is practical, feasible, and justifiable to develop one instrument 
which could be used to assess degrees of implementation across all 
programs, standardized and individualized, and across all sites; or 
b') whether a unique instrument and set of procedures should be developed 
for each t)f the* programs in order to assess in a reliable and valid 
way the degree to whiVh a specific program was being implemented across 
indiA^idual classrooms utilizing that program. The implications for data 
connection and analysis of the -two alternatives are rather significant. 



First, if one couM develop a uniform instrument for assessing 
implementation across programs and sites, and if such an instrument 
v/ould provide reliable and valid data for use in sample selection, 
verification, and data collection during tbe period-'of observation, 
then the data col lected. could be used ih the analysis aS an independent . 
variable which could possibly be associated with' several outcome or 
dependent variables across all programs. 

Second, if one were to develop unique instruments to assess implemen- 
tation for each program, the leu^k of a common instrument woul(l erode 
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-'somewhat the .validity of assfessing-degree of implementation across 
. programs. However, information gathered through surveys, program audit 
checklists, and observation, based upon prescribed implementation guides, 
would probafelfbe more reliable and valid with respect to the degree^ 
to wKich a specific program was being implemented according to a pre- 
determined plan. While the" instrument in'the former alternative would 
have to be global in nature, the procedures developed under this alterna- 
tive would be less obtrusive (i.e., through the review of documented 
evidence such as student. progress check records); observations could ^ 
be reduced; thereby avoiding anxieties on the part of teaching staff, and 
would probably be less time-consuming than the observatiorrs which 
•.occurred during the SRI study. * 

Both of these alternatives Kave several inherent problems. First, 
since the final program sel ec'tion wil 1 not occur until the spring of 
.1976, no general instruments for the selected programs could be done 
with certainty. (However, as a surrrogate, one could select programs 
from potential candidates and assess their respective implementation 
procedures.) Second, discussions with Follow Through staff at the 
respective Follow Through modql development sites 'indicatelhat, with the 
exception of the LRDC model, most programs are undergoino/revision at 
the present time. Hence, revisions of the instruments /ould probably 
have to be made during the implementation phase of th/s study. Third, 
dependence solely upon imp'lementatioii guides suppli/d by publishers 
rather than reliance upon actual implementation pdns as modified by 
LEA staff is rather tenuous. Based* upon project/team knowledge and 
experience in a large number of LEAs where in/ vi dual i zed learning 
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are presently being used, the adaptation process has frequently^ resulted 
in major changes in procedures. For example, in Grand Rapids, token, 
economies are used extensively in programs where they are not necessarily 
suggested or prescribed. In Dallas, instructional programs have been 
modified to accommodate the district-wide criteria-referenced testing system. 
" In short, while publisher implementation guides, provide a starting poiht, 
verification will require discussions with LEA staff, at which time the 
possibility of revision of instruments must be assessed. In order to 
select either of the above a.1 ternati ves or some suitable combination,* 
project staff members spent an extensive amount of time reviewing 
existing instruments and studies and conducting discussions with the ^ 
individuals noted above in Appendix 8. From these discussions the follow- 
ing findings and observations appear to be relevant. 

I 

Findings 

1 . Existing instruments used t o assess degrees of implementation 
are inadequate in terms of comprehensiveness, appropriateness, and 

degree of reliability, and validity, and adaptability for the purposes 

of this 'Study . • ^ ' 

The specific instruments and procedures -reviewed for potential 

application adaptation included: the /I/D/E/A/ IGE implementation kit: the 

SRI Follow Through instruments; and 10 individual implementation guides 

■f 

and checklists recommended or used by publishers* consultants and/or 

LEA staff in assessing the degree of implementation of their .individualized 

learning systenis* 
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I^E Implementation Kit ^ _ - 

Over the last few years, /I/D/E/A/ Kettering staff assigned 
to monitor the 'iinplenientation of IGE throughout the country have 
developed an implementation guide or kit which specifies 35 out- 
• comes that become the central focus of continuing in-service and 

implementation. The instrument is rather comprehensive in assess- 
irlg process objectives which should occur during the planning 
and' implementation of IGE programs over a period of tinie. Based 
on discussions with /I/D/E/A/ officials and a review of ^evaluations 
sponsored by /I/D/E/A/, it would appear that th.e items are appropriate 
for the IGE program. It should be noted that IGE is generally 
a' large network, consisting of a "league" of schools which agree 
to certain conditions prior to adoption and implementing IGE. Also, 
the IGE process focuses upon rather global processes related to 
c-u^^T ^v-,^->oi*T:>f ■Jr.rt -inx/ni \/omont nf n;^rpnt<; and the coiTiniuni ty > 
■ and staff development in a variety of individualized activities. 
The implementation checklist does not focus upon the implementation/ . 
procedures of a specific program: rather, in many cases, an 
individualized instruction progra;ii such as IPI, with' its pre- 
scribed procedures, will be implemented in an IGE school. In 
certain instances, conflicts exist between the IGE processes and those 
•suggested in structured individu^al ized packages. The IGE implemen- 
tation kit provides for a number of proc^esses which appear to be 
•appropriate for assessing implementation in this study. \H°wever, 
for the most part, the instrument is limited to surveys rfcther 
than observation, does not focUs specifically upon the classroom * 
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'implementation procedures, and^'s structured around the IGE approach^. 
SRI Follow T^irough Instruments 

The recent SRI/ollow Through study represents the most 

intensive attempt to assess program implementation that has ever 

been conducted, although "differences in implementation". was only. 

one of the concerns addressed. The'survey and observation instruments, 
» • * 

developedHbroiigh a ri go rous" process, were designed to answer a 
^number of other questio^i^ in addition to program implementa-tion. 
Once these variables, including the implementatipn variables, were 
sdeveloped/an attempt was mad^ to identify (through fonsultation 
with the model developers) the specific ^implementation variables 
^vhich appeared to be critical to effective iuipl ementation of the 
respective Follow' Through models. The unit of comparison was b.ased- 
upon the implementation procedures used in a '^standard classroom." 
Hence, rather than .determining how v/ell each Follow Through n^odel 
site implemented the Follow Through model program in accordance 
With a pre-determined plan, the study attempted to determine what 
differences, if any, existed between implementation procedures of 
Follow Through when compared to a. standard classroom program. 
In addition to extensive classroom observation, a teacher' survey 
instrument was also used with Follow Through participants. Re- 
suiting data, however, was found to be inadequate for analysis- ^ 
purposes. 

As noted earlier, the Follow Thr(^gh instruments (mostly * 
observational) v/ere not comprehensive enough, although some attempts 
to gather additional information were made. For example, parental 
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involvement, as part of the Follow Through requirements, -was 
observed o,nly if parents performed services as a participant in 
classroom activities. IN cases where ''at-home" instruction was 
provided in a nuinber ^o'Y^the Follow"* Through models, observation's^ 
were not conducted. Perhaps most importantly, while attempts 
w€re made, data was not available to^ determine whether differences 
in project preparation activities prior to the implementation 
of the programs occurred (e.g., planning^ training). In the Michigan 
Cost-Effectiveness study, TURNKEY found that a number o^ variables 
such as the decision-making power over selection of materials 
delegated to teachers ,* the nature of teacher training (e.g., in 
diagnosis and deygiopihg performance objectives), the extent 
of itraining prior to program implementation, and other planning 
activities discriminated significantly between successful and 

M r\<* • • O ^ ■^t • T r> v« »^ -* r>"^ A r» \/ -1 «^ r mii t r» r\ 4- r> Ar\f'^nr\^kA -t' r\ o + f\ *-»iyi 1 r» o fKr» 
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degree of implementation must also take fnto account the pre-inipleaien- 
tatfon activities which have occurred, the qualitative nature of 
these activities, and to the extent possible, the relative impact 
on effefctive implementation. 

Eyen through the procedures used in developing the Follow 

f 

Through; instrumerits were extremely rigorous and were determined 
to be reliable through extensive training and inter-rater reliability' 
checks, a serious question has been raised by a number of individuals 
involved' in developing and providing technical assistance to Follow 
Through sites about the validity. of these instruments. In addition, 
some of the unexpected findings (e.g., the negative correlation 
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between praise^-and student performance) also raise questions 

regarding the validity of data collected through observations. One 

Follow Through model developer indicated that several errors of 

omission occurred -in the sense that the number of implementation 

variables provided to him for selection and prioritization (e.g,, 

critical vs. important) did not reflect certain activities which 

were uniquely critical to the model. other instanfces the 

variables 'from which the selection had been made did not adequately- 

describe the specific manner in which the activity was to have been 

conducted. In*at least five instances, lEA or fnodel development 

directors who were involved in the SRI Study did not want to be *. 

considered as a candidate ^si te for this study due to their "bad 

t 

experiences" with the Follow Through study. In light of the great 
care taken by SRI, the extent that the Follow Through instrutnent 

to have occurred) accentuates the difficulty of developing a single 
instrument that can be applied uniformly across programs which 
have different implementation plans. 

• Selected Speci fic Implementation Guides/Checklists 
At this writing the project team has reviewed ten pre-implemen 
tation guides (e.g., teacher guides, supervisor guides, and ad- 
ministrator guides) which are included either'in a pre-service 
training program or as part of the overall information package 
for an individualized learning system. The specific projects vihose 
guides have been analyzed in a preliminary njanner include: 

IFl ' , , 
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a) Project TEEM, University of Arizona Follow Through Model; b) 
New -Century (formerly Appleton, Century, Croft) Math and Reading • 
System; c) the five program i,Riplement.atior> components of the five 

^ Program Information 'Packa^s (PIPs) presently- being field tested 
by USOE; d) High .Intensity Learning System (Cohen/,Random House 
System); and' e) SRA Individual,ized Math System. 

' Feasibility Testing Instrument Design 
Based upon"" a review of these program guides, the project team de^veloped 

a list of general categories of implementation variables. In addition, 

. - / • . ' 

the various types of data collec^on instruments feasibl^uf^n collecting . 

data on the specific categories of implementation variables was also » 

identified. These variables are presented in Exhibit 111-^15 which also 

identifies the -type {e.g., interview) and purpose (verify vs. initial 

collection) of data collection instruments which could be used. 

U^n identification of these variables we designed a preliniinary 

observation instrument, following a format similar to that used by SRI 

in the Follow Through study and by Medley in devising the PROSE and OSCAR 

instruments. Basically this .instrument identified "who" (e.g., teacher, 

student, teacher)^ how" (e.g., verbal, written* instructions, etc.); 

and "modeirators" (e.§., praise, dispraise, neutrality) as indicated in 

Exhibit 111-16. The answers to such questions were designed to assess 

events and to describe observed. interactions and classroom cictivities. 

We were concerned, -however, that such an^ instrument would lack appropriate 

validity and discrimination qualities. We, therefore, conducted a 

number of simulations and limited classroom observations to determine 

/ » V '* I » " ' 
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X 


X 


X 


, C. Instructional Planning 








1) Role of Individual or Group Instruction 'within the 
System \ ^ - 1 1 3 
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f 

X 


X 


X 

• 
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• 


* ^ 

• 

* 


RfPES OF - 
INSTRUMENTS 
\Tlia4 COULD BE 
JSED 


G\TEGORir:S OF ACTI\aTIES/E\'TENTS, ^ • 


IP 


OBSERVATION 


E- 

CO 

t— < 

us 
u 

a 

u ' 


II. .PROJECT 0PBR.A.Ti6n? {Con^-cl) 


* 








Instructional Planning (Cor^^M) 


• 








. 2) Media'^of ' instructional, planning. ' . ' * 


X 

• 


X 


x' 


D; 


Instructional Management : ' ^ , - 

f 










*1) Student Recojds/ Information Retrieval 




X 


X 




2) Inforaation Sources Used for Instinictional Decisions 


X 


X 


X 




3) Ngture/Frequericy of Decisioib 


X 


X 


X 


H. 


Classroom Management ' . - * ' 


■ 








• 1). Usa*bf Reeources 


X . 

* 


X 


X • 




2) 6r.gani2ation of Learning Environment 




X. 


X 




3) Student Nbvemcnt in Learning Environment 

> 




X- 


X 


F. 


^ Student >!otivation ' ' . 


X 


X 




• III. PROJECT EVALUAHON 








A. 


Jnvolvement of Students/Staff /Pa rents in Evaluation* 


. X 




X 


• B. 


Types of Evaluation Recommended for Project 


X 




X 


C. 


Data Collectioix ' * « . 


Y 
A 




Y 
A 


' D. 


Anglysis 


X 




X 




Reporting ^ 


X 




X 


F. 


Use of Evaluation. 


X 


x 


X 




.1) Feedback into the System 


X 


X 


X 




2) Other . • ^ ' ■ • 




X 


X 


* 
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Exhibit II I -16 



Obsen^'ation Design 



mo 



MiM (ACTIVJ-nO 



TO nm 



f Teacher ^ 

A Aide 

V Volunteer 

S/ Student ^ 

G Sma^l Gyoup , 

L' *Large Group 

P Principal 

R Reading Specialist 

C/R Computer Readout 



. D V Diagnosing 

' * A/|P Assigning/Prescribing 

Q Questionning 

A Answering 

'L Listening - 

0 Observing 

R Recording 



T 


Teacher 


A 


Aide 


V 


Volunteer 


S 


Studeht 


G 


Small Group 


L . 


Large Group 


P 


Princ ipal 


R 


Reading Specialist 


C/R 


Coniputer Readout 



*HOW 



V , Verbal 
W Written 
D Dranja^ 

P Pagehtry (Physical 
non- language') 



V 

^DDKIlATORS 

P Positive 
N Negative 
0 Neutral 
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whether or not the recdrded responses woifld adequately assess the 

'qualitative nature, of "degree af iriiplecientati on,/', recording .in narrative 

' form en'ther the actual practice observed in th.e 1 imited classrooms or 

suggested procedures described" in ifnplerheotaticn .guides. Exhibit'Iir- 

17 describes the result of the-se," field' test" activities. ' • _ . 

Suggested agd Optional Procedures > " ■ . i ■ , - . 

. . ■ ; ■ ••„''".•.' 

In light of the limitations of 'the above approach, "we have refined 

the, second alternative — the use'.of a" specific i^jumeitt&^Wd implementa- 

' tion cri.te^^ia'for a'ssessing'.each ,learni(^g;sy5lte(rt_u3ed" -r .aiid designed 

some preliminary instrumejits to-Hltustrate.the 'appro^ich. In addition, 

we developed an optiocia-l procedure and instrument, tl^e feasibility of 

which will have to be tested during ^he implementation phase, to assess 

among programs the degrees of implementation only, along the dimensions 

of individualization specified in the. sajiip^ling^de"sjjtT^(e.g^^ the 16-cell 

matrix). A major differor.co bctv^ccrr the suggested procedure and the 

optional procedure is the degree MMnvplvement of the program developer 

in determining the -instrument whi'c-h Wto evalute degree of implementation 

along with specific dimensions. rWith tResuggested procedure, the 

developer has the major responsibility for defining the questions and 

observations within categories sjiecified by the contractor. If the 

- optional procedure is used, th6 developer is more limited in determining 

questions and observations, buti still must define the range of acceptable 

answers and verification procedure«^swhich could be unique to his pjfogrm 

a procedure similar to that used by SRI in'tBe Follow Through study. 
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Bd^ibit riia{7 
Pdge. 1 'of .7" ■ , 



QUASI FIELD TEST OF OBSERVATION INSIWENT 



The .following is an excunple of the problems which occurred when tiying to use 
a imiform observation checklist on'^five different existing learning, systems 
which were in operation in potential target sites. The five. programs ' ; . 
considered ha\-e all met the eligibility criteria discu5sed,in Task 2. These ^ 
programs are listed! below according to the following notation,: ' . . 

An Alpha II program in Grand Rapids, Michigan 
A Learning Unlimited Program in Grand Rapids, Michigan 
A Kestinghouse PLAN proorain in Grand Rapi'ds, Michigan 
A Ginn 360 program in Grand Rapids, Michigan 
A Middle Cities program in Lansing, Michigan 



ALPhl^ 
LU 

PLAN 
GINN • 
M.C. 



CO 

An area of the activities/events listediin LxJiibit riI-16 which were basic to 



'all individualized programs was selectejci. 



/. 



II. Project Operation 

-A. Diagnosis/Testing 
The activities were recorded on an observation fonn. 



Activity 1: 


Stud^eiiL 


Tdkob Diagiiu;>ib 








PROGRl^l 


WIO 


IVllAT 


TO MIOM 


HOW 


^DDE1UT10N 


/\I.PHA 


T 


D 


S 


w 


0 


LU 


T 


D 


S 


w 


0 


FLVN 


T 


D 


s 


w 


0 . 


G\K>i 


T 


D * 


s 


w 


0 


M.C. 


T 


D 


s 


. w 


0 


Activi^tju^r 


'^Results of Diagnostic Test are 


Recorded 




' 'PR£X]IWI 


WIO 




TO MIOM 


HOW 


M0DER^TI0N 


ALPllA ^ 


^ A 


R 


S 


W 


0 


LU. J 


A • 


R 


S 


W 


0 


PIAN 


A 


R 


s 


w 


0 


GINN ' ' 


T 


R 


s 


w 


0 


M.C. 


A . 


R • 


s 


w 


0 



ERIC 
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Exhibit III-17 
Pag§- 2 of 7 



Activity. 3fe Student J^eceiVes .Prescription Based on Diagnostic Test 



■• PROGRAM 


,*, ' ' ^vyo 


' ' 1\HAT 


, . ■ TO WIOM 


HOW , 


NDDERATION 




T , 


A/P 


S . 


W 


Nu ' 


• uj • 


• C/P 


A/P 


s 


. ii 


Nu 


PLAN 


C/P 


' A/P 


s- 


•w 


Nu ' 


GINN 




A/P 


s 


'v 


Nu 


M,-C.' 


T 


A/P 


, s 


w 


Nu 



Using this't>'pe o'f observation format, little difference was noted bet^veen the 
various pro£;rajns . 



B^: interacting with site staff and program developers a^ reviewing implementat 
guides, it. was determined that major differences did, in fact, occur in this 
simple diagnostic-prescription step, Tliese differences would be important to 
note between programs, but more critical they must be noted by those responsible 
for ensuring that effective implementation of a pa:'ogram has occurred. 

The following scries of diagrams depict signific^mt differences in the observed 
piugidjiib which were not identified by a simple cross program observation 
checklist. 



S^^BOLS 
FOR 

FXVNPLE 



A = Assigned Material 

/ = Test \vhich is not used for Diagnosis (changing thfe students 
assignment) but only to ensure. he is accurately resix)nding 
• to tlie material 

T = Test which measures an objective but does not change 
assginjnent\ . ^ 

D = Test which changes an assignment (i.e.. Diagnosis for 
Prescription) , 
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Exhibit IIM7 
Page 3 of 7 



ALPHA Block approximates a 2-week work assignment 



Objective 1- . Objective 2 






, A 


1 — ^ 



— (D 



Objective 3 




Small objective units * intcr-modual branching pbssitjle. 



PLAN 



TLU approximates a^2 4veek assigmnent 



uu jective • 



A 




A 




• A 




A 


1 












No small objective units. Branching to D test only if student "feels'* he wants 
the '^challenge'*. v ^ . « ' 



LU 



^todule ^pr6ximates a 2 week .assignment ^ 



A 




A 




ERIC 



Small check tests - not used for branching due to "spin off learning. 



> 119 

III-70 



'Exhibit IIM7 
Page 4 of 7 



GINN 



BOOK 2 weeks m 1 year 







Instruction 



' Ii>struction 



Instruction 



Reading assignment does /not change after skill tests - only Tecommended 
instruction by teacher i)r assignment of supplementary dittoes. Assignment 
Book not cbnsistant with skills tested. 



M.C- 





No sequence of tests required. Primarily evaluation with "available" cross 
referenced materials. All instruction noted accomplished by paraprgfessionals , 
outside of regular class environment. , 



The maior differences between programs can be specified only when the frequency 
of the diagnostic testing is detennined and the consequences of the testing. In 
one system (LU) daily tests are used to ensure that the student is responding 
to the system. Withigood classroom control some site personnel feel they could 
be pmitted mad^ie system could be considered well- implemented. In-a second 
syst^K(.UPfL\)d tests axe. used for branching and are critical to the 
systcm.^-li'iese tests could not<bp oniniitted in a well -implemented center. 



er|c 
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Exhibit III-17 
Page 5 of 7 



In a third system, PLAN, no daily testing, exists for either diagnosis" or student 
control. All testing is diagnostic,"" but occurs after major tl-2 week) time 
intervals. In still a fourth system, GINN, tests may occur daily or weekly but 
wili not modify the reading assignjrient and may be given at such times when a 
teacher is planning new instmctional skill groups. / ^ ^ 

In each system, testing is obser\'abl]b in the classroom. The consequences of 
the testing require not only that ^e obsen^'er interview the- teacher, but also, 
that the obser\^er be aware of the program developers purpose and intent for use 
of each of these instiTiments.. 



^Based on these differences, an interview fom was attempted. An additional 
problem soon becaine evident. Wilh an inter\dew form, project differences 
could be detemincd. But, w^erc these diffei*ences critical for the implementation 
' of a specific project? . , " 

^ ^ 1/ 
' The following list of questions are a selection of those that' quickly demoiis.trated 
differences between potential programs. Ten of these questions are listed heTe 
for purposes of denionstrktion. In these questions the word '^module'* is b)nonomous 
with Block .(.UPHX), TLU (PLAN), >DDULE (LU>> Reading Assignment (GINN), 
, teacher assignment for an objective (M.C.). ' 

VuT.STIOXS Ivli'IQi DISCRIMIIvuTD BEn\T>E\ FROGR/v^lS OF vVvRIOUS 
OXTEGORIES: 



INTERVIEW QUESTION 



R/\TI\G OF 'n'OTALLY 
IMPLENILNTEDy CENTER 

ALPHA- ""lIJ 1pLA\ GL\^^ M.C. 



1. Does student take a test which will allow for a 
Branching decision on a daily basis? 

2. Within a ''module**, can a student indicate 
objective mastery? 

3. Within a * 'module'* can a student work on onl)^ 
those skills necessary to complete that module? 

4. Does a- student usually work in more than one 
instryctional modeule in a given "area at a 
given time? 

5. Does system allow teachers to detemine if a^ 
student is functioning in the system and 
accomplishing assignments successfully? 

V 



Yes No No No Yes 



Yes ■ . No No Y.qs Yes 



Yes No No- No Yes 



Yes Yes No No Yes 



Yes Yes No' No No 
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ExKibit III-i: 
Page 6 of 7 



QUESTIONS- WnCll' DISCRIMIN..\TED BETls'EEN PROGRAMS OF VARIOUS * ' ' 
OXTHGORIES:' . (Cont'd) 

RATING OF "TOTALLY 
. ■ IMPLENBNTED" CEMER 

IM'ERVIEW QUESTION " . ALPHA TlU^ "pLAn" GII^sI M.C. 



6. 


Does system allow daily indication of student 
.work effort? 


Yes 


No 


No 


No 


No 


7. 


» 

Does. system provide systematic review of student 
performance to administration? 


No 


Yes 


Yes 


No 


Yes 


8.' 


Docs system provide systematic review of student 
performance to parent? - \ 


No 


Yes 


• Yes 


No 


No 


9. 


Does system stres^S'teaeher presentation of ^kill? 


Yes 


No 


No 


Yes 


^No. 


10. 


Does system stress use--of incentives in 
classroom? 


Yes 


Yes 


No 


No 


No 



If we use these 10 items with equal weight in^? for each (yes^l no=0) to 
is obtained: , 

ACTUAL SA^1PLE R-VflNG OF CENTER ON TEN INTERVIEW ITF.^IS: 



ITEM 


/\LP11A - 




PLVN 


GLN'N 


M.C. 


1 


1 


o' 


0 


0 




2 


' 1 


0 


0 






■ 3 


1 


0 


0 




?\ 


•4 


1 


1 


0 


I 1 




5 


1 


■ 1 


0 




0 


6 


1 


0 


0 




0 


7 


0 


1 


•1 




) 0 


8 


0 


1 


1 




0 


9 


0 


0 


0 


0 / 


0 


10 


0 


0 


0 


C- 


0 


Percent of 












Implementation 




m 


20? • 


. 1D% 


40% 















It becomes obvious with this examj^le, that the implementation in an actual project 
site can not be detemined by a standard implementation interview fonii. 
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Exhibit III-17 
Page 7 of 7 



Tlie* degree of iniplementation must be determined by comparing the "theoretical*' 
for each- program to the ''actuar* for each program site. If this is done with 
this particular series of project observations, a totally different profile for 
ijTTplementatioa is obtained. 



TOTALLY IMPLT3IENTED PROGRV^IS: 
V = Variable's relevant 
- = Variable*^ IS not relevant 



ACrUAL SITE OBSERVATIO>{& ON PROJECT: 
(ib^vant variables only) 

'doesn't 





ALPHA , 


LU • 


PLAN 


GINN 


M'.C. 




ALPHA 


LU 


PLAN 


GINN 


xf C 
i*j • o * 


1 


V 








V 


1 


1 








1 




V 






V 


V 


• 2 


1 






1 


1 


3" ; 


V 








V ■ 


3 


1 








1 


4 ' 


, V 


V 






V 


4 . 


1 


1 






1 


5 ■ 


V 


■ V 








5 


1 


1 








6 


■( I 










6 


1 










7- 




V ■ 


V 




V 


7 




X 


1 




0 


s 




V. 








8 




1 


1 






9' 


V 






V 




9 


" 0 






0 




10 • 


• V 


V 








10 


0 


0 






















"6/8 


4/5 


2/2 


1/2 


4/5 
















75% 


80% 


100% 


50% 


80% 



In completing this analysis, ve selected those questions which did discriminate 
between sites. In a tetal interview form, these questions would not make the 
dramatic difference that they did with this exaTiple. However, the difference 
would be great- enough to invalidate any attempt to use degree of implementation 
as a variable which might itnfluence program outcome. 

To erisure that critical* variables to show up in the filial analysis, program 
developers will assign weights to all activities and processes in observation 
that they -specify as a part of the implementation procedure. . 
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A, Suggested Procedure 

The same general categories of implementation activities (Exhibit ' 
III-15) which would be common to virtually all learning systems would 
be used to determine the degree of , implementation of both individualized 
and standardized p)?^grams. ' ' . . 

The ^Process which would be used upon final selection of the can- 
didate sites would resemble that used in the SRI Follow^Through study, 
with the exception that responsibility for including and/or expanding 
specific activities and criteria for assessing these activities wtII be 
shared with the model developers or publishers. Based upon a review of 
existing implementation procedures and observation instruments used 
for monitoring implementation, specific activities would be listed for 
each c'ategory and tentatively weighted as "critical", "important", 
or "not important". Those specific activities, procedures, and assess- 
ment criteria will cans^tute the data requirements for instruments to 
be used to verify the extent to which a potential' candidate is well- 
jmplemen.ted, priqr to final selection and inclusion in the study. These 
data requirements for instruments will be included as an addendum to 
the questionnaire to be sent to the appropriate LEA officials, as described 
in Appendix' 5.^ Additional telephone and on-site follow-up. will be 
required to complete the verification process regarding the critical 
activities and fevdnts relevant to the operation of the program during 
1976. 

In Appendix 11, we have identified anumber of critical events 
and activities for a sample potential candidate site. Using the general 
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categori.es in Exhibit III-15, .the candidate site staff and consujtants 
will have lis,ted and weighted the specific items and criteria which will 
be used for evaluatiJig implementation iri their site. In. this example, 
the data requirements and assessment criteria reflect the idiosyncracies 
op implementation procedures inherent in this particular program. 

To develop the specific requirements for implementation it will 
become necessary to not only involve the publisher or model developer • 
but also the most knowledgeable' indiv.idual(^) at the sites to determine 
the specific ways in which the programs "should be" implemented. The 
specific items which must be agreed upon will .'include: (a) ah explicit 
description of classroom procedure's (e.g. , diagnosis , prescription) 
and alternatives which are acceptable for well-implemented programs; 
(b) a relative weighting assigned to each of the items or observations; 
(e) -specific criteria indicating the accomplishment -- ^or lack thereof — 
with respect to each of the activities or procedures (e.g., percentage 
of materials which are observefl to be available ^nd accessible); and 
(d) the identification of, source documentation wherever possible. 

The manner by which. mutual agreement wll be sought will differ 
to some extent from site to site and from program to program^_^pending 
upon the complej<ity of the implementation procedure and, existing doc'j:r.en- 
tation. However, for the most part, it is anticipated that the project 
staff will specify the -categori es (as shown in Exhibit III-15) and con- 
tribute to the basic design ^characteristics of the instrument. FinaViza- 
tion of the instrument will reqjjire discussion and joint agreement between 
the project staff and the program' developer at the potential site. Siiice 
it is the desire of both the project staff and the program developer to 

IPS) 
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include well -implemented programs, disagreement on ^the instrument should ' 
be minimal . ' " 

Once the data requirements have 'been specified with respect tO' the 
procedures and assessment criteria for each of the programs, a determina- 
tion will be made regarding the specific type of data col lection, in- 
strument to be used. We anticipate that the types of instV'uments will 
include: interviews with building and district staff, audit checklists, 
and observations. The purpose of including items in interview fonris, as 
well as audit checklists, will be to collect initial data and to verify 
this |data through review of^evidence. The major purpose of observation 
will pe to verify the existence of specific activities. In addition 
to relatively structured interviews, opportunities for un5;^tructured 
interviewing will also be available as a follow-up to observation. For 
example, observers will be trained to' identify critical activities and 
events. Upon ths occurrence of such activities (e.g. , the teochGr 
prescribed a specific learning activity for an individual student), 
the observer ^{at the completion of the class period) would ask the teacher 
why she/he assigned that learning activity to that child. The observer 
* will not only record the teacher's response, but also identify the specific 
instructions in the teacher .guide) . 

While the general procedures for integration during the overall 
•data collection and analysis activiti^es are desotib^ in Task 5, the 
implications for data analysis, utilizing this/approach, are important. 
First, the abo-ve procedures will ensure rgli^ble and valid data regarding 
the degree to which a potentially v/el 1-implemented program is actually 
being implemented. Through scaling prooledures and the use of mutually 
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agreed upon weights, onfe will be able to report upon .the degree to which 
discrepancies occur in important implementation categories. Within a 
particular program it is anticipated that analysis will provide useful 
insights regarding the degree to which implementation occurs in specific 
classrooms. Second, to a much lesser extent, one will be able to 
identify the degree' to which di screp'ancies occur between programs in 
general category areas and assess the impact of the variables as independent 
program variables upon dependent or outcome variables. Hov/ever, it is 
expected that some judgmental decisions will be needed in^assigning 
values to the expected differences in critical variables. Third, this 
approach would allow ^ to capture more relevant information on the 
qualtative nature of implementation activities as they vary within the 
program and within the classrooms. Discussions with publishers and model 
developers always tend to "emphasize" the qualitative nature erf imf)lementa- 

provide the opportunity for identifying not only the existence of barriers 
which "^impede effective implementation but also the justifications provided 
by respondents. While the 9ollection of anecdotal data in this area may 
be difficult to quantify, the coljection of such information has assisted 
enormously in past projects involving project team members (ev^^^ the 
USOE Project in the Use of ' Incentives) and explained other findings which . 
emerged from hard data analysis.. 
B. Optional Procedure 

As requested in the RFP and subsequently by the NIE Project Officer^ 
we have devloped an optional procedure and instrument for assessing 
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implementation across programs only with respect *fo those dimensions; " 
' of ' individualization on whoch the programs are designed to differ. The 
feasibility of*using this j'nstrjjment will be tested early during tiie , 
implementation ^phase. If'f^llsibfe, it could^^ integrated into the 
"suggested" procedure previously described and would have the advantage 
of allowing for cross-program comparisons. 

The primary purpose of this set of procedures and instruments would' 
be to assess programs during their operational phases. Hence, al > categories 
described in Eochibit III-15 under Program Operations would be collapsed 
•and grouped into the four djmensions of indivi d,ual ization; (1) performance 
.objectives; (2) continual diagnosis and prescription; (3) student pacing; 
and (4) ajternative paths of instruction. Even though these dimensions 
are considered as discrete components in Task 2, it should be noted that 
operational activities in these four areas overlap to some extent. 
Moreover, since, predetermi ned sets of implementation variables will 
have been defined' (as in the SRI Follow Through Study), the need for greater 
interaction with LEA staff, and possibly. developers/piJblishers , will . 
be even 'more critical in obtaining accurate information and assigning- 
weights to specific, variables (Discussion with SRI staff indicate the 
time-consuming nature of these interactions.) The suggested procedures 
and steps to be foVlowed by the implementation contractor are similar 
in some respects to the suggested procedure described above. 

First, after initial selection, an on-site visit will be required 
to verify specific dimensions of the program at the site and to 
determine the relative importance of, variables under each of the dimensions, 
w^ich are appropriate for that program. It is probable that the implemen- 
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tation contractor will have to meet with a number of LEA staff . involved 
in developing the program and/or training teachers for its implementa- 
tion. * 

4 

^'cond, duriag the on-site visit, an instrument such as that 
suggested in Appendix 12, will be reviewed with the appropriate lEA , 
staff, who will: (dj identify those" variables which are not applicable . 
(N/A) to the program; and where applicable (bj indicate the acteptable 
range of response (or^ observation); and (c) assign a weight to implementa- 
tion variables. In order to re'duce respondent biases, it is "important ^' 
that the initial responses be completed for all items prior to having 
.k'h'owledge that weights will be assigned. Also, in describing the purpose 
of the study, it should be made clear that the major, focus is upon 
the outcomes of v/el 1- implemented programs, thereby implying the ad- 
vantage of narrowing the acceptable range as closely as possible to 
the prescribed in^plen^sntation plan. 

Third, wtTe re possible; the implementation contractor's staff, 
while on site, wi*ll administer the instrument in a limited number of 
sites, in several classrooms, to determine its feasibility. It is 
anticipated that two or three staff members involved in the refinement 
of , the instrument, or one similar to that. in Appendix 12, will 
conduct these observations, insuring some degree of interrater reli- 
ability. ^ 

If the above procedure and instrument are feasible and provide 
accurate and reliable data, then, upon reyfew by the implementation 
contractor and NIE, a decision will 6e made as to whether or rfct to 

r 

« 
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us6 this instrument in lieu of portions of the instrument described 
under the" suggested procedure above. The* major advantage of this option 
is, of coi*rse, thai cross-program comparisons regarding degrees of implemen- 
tation could be made, possibly relating them to outcome measures. It 
should be noted, however, that implementation will be assessed only 
along those dimensions of individualization by which the programs are v 

designed to differ. In addition, aside from information regarding ^ 
• , • - ■ • *«•■'* 

planning, teacher training, and project start-up, the differences* 

.regarding implementation between individualized and standardized programs 

* ■' 

will be significant since the dimensions to be assessed are tho^se re- 
fleeted in individualized programs. ^ > . ■ ' ' . 

We have included the above optional procedure and instrument- . - . 
in direct response \to^ request from. the NIE Project Officer. We are 
still, however, very concerned regarding its feasibility and utility. 
This feeling has been buttressed more recently in discussions with 
additional Directors of Follow Through models which were included in 
the SRI study, and other knowledgeable consultants, including 
Dr. Richard Rossmiller at the Wisconsin Research and Development Center 
for Cognitive Learning. ^ * 
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TASK 4 - DEFINE OUTCOME BEHAVIORS 

Given the mul ti'faceted character of educational goals, three general 

categories of criterion variables will be incorporated within the design of 

this study. The first, academic achievement in reading and qiathematics, 

requires no special justif ieation since it is a. generally agreed upon 

priority in compensatory education. In addition, while achievement in both 

areas is strongly related,* reading scores at^the Znd^and 3rd grade level 

tend to be' indicative of achievement in all the contents areas at the junior 

and senior high level. Th.is point has been supported by the research of-^ 

Bond and Dykstra (1967) and Neuman (1972). The two other areas to be con- 

sidered in the design are: (a) stude^nt and teacher attitudes toward the 

•school environmen-t; and (b) student self-concept. Averch, et al., (1971) 

have critic-ized th£ measurement of limited school outcome variables within 

educational research stating that: 

'^educational outcomes are almost exclusively measured^P^ 
cognitive^hievement. But the educational system h3sr\.many^ „ ^ _ 
functions and many outputs. Cognitive achievement, in 
pari tctilar Ihat part measured by standardized tests, is 
only one aspect of student learning (p. 153)." 

Both of these affective areas are seen by many as. having important 

educational valuB because, as some educators believe, development of student 

affect is an important end in itself (e.g.. Holt, 1964) and has serious 

implications for studpnt success in future academic endeavors (Bloom, 1964). 

Such affective characteristics of both students and teachers may help to 

modify learning outcomes by directly affecting the amount of time available 

for effective learning acquisition and by affecting student motivation 

towards learning. These contentions are supported by the extensive research 
r 

4 

efforts in this area (e.g., Brookover, Thomas and Patterson, 1964; and 
Dryer and Haupt, 1966). Thus, in order to evaluate educational programs 
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and- their effectiveness for promoting student learning, the design 

must focus not onlV, on students' achievement but also on the qualities'' 
•of the educatipnal environment that are though likely to promote such 
achievement. This approach provides the opportunity far identifying 
potential factors which may help to explain differential achievement across 
programs. 

In assessing outcome variables, only those instruments which provide 

an objective diid valid means for assessment and scoring will be' considered 

for use. This reduces the need to use observational procedures as a means 

of assessing outcomes. The major arguments against observational systems 

and procedures to measure, outcomes is that they suffer from many metholo- 

logical flaws, including: (a) reliability problems due to lack of interrater 

agreement and instability of the behavior being observed, and (b) difticulities 

fn accurately observing the actual behaviors of interest due to the ob- 

trus]ve effects which observers have on the individual and/or the environ- 

ment being observed, and sometim^due to the actual nature of the variables 

of interest. Herbert and Attridge,^ as referenced in the RBS proposal, 

point out that the work which has thus far been done in the identification 

and solution of the methodological problems of observation is inadequate 

to ensure accuracy of the findings.^ 

A Rationale for' the Use 9f Standardized Achievement Jests for 
Assessing Academic Skills * . 

If one were to think of individualized and standardized programs as 

differing only with respect to instructional procedures with exactly the 

same objectives, the task of comparing the effectiveness of the two 

instructional approaches would be relatively easy; build tests tailor-made 
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to the programs' common objectives, administer them,, and conjpare the results 
across programs » However, the very nature of individualized programs 
delimits the effectiveness of using a single tailor-made test since the. 
objectives of specific programs often differ or, at the very least, may 

be prioritized or sequenced differently across gra^ levels* Thus, no single 
tailor-made test can possible be equally fair (or equally comprehensive^' 
or complete) for all programs. ' ; . 

Rather than using many specific tailor-made tests which prohibit 
comparison, it will be more appropriate, to use a sing,le, more general, 
achievement measure such as one of the "l-tand'ardized norm- referenced in- 
, struments- These instruments are nW designed ^o measure a narrow set of 
specific objectives that might'be found to exist for $ny given program, ^ 
but are designed, according- to pubj j/shers of such tests, to assess achieve- 
ment in a fairly diverse area. Contrary to the writings of some uniformed 
critics of such general tests, they do have some content validity as they are - 
(in many cases) designed to measure a broad "national curriculum". This 
"national curriculum" is identified by carefully surveying and assessing 
curriculum experts (as to what the curricula should contain), the diverse set of 
curricula found in various educational systems, and curriculum textbooks 
and other materials which are receiving wide use in educationa? institutions- 
There is no prior reason to believe that such tests more closely match thfe 
reading and mathematics objectives in "standardized" programs than they 'do 
the^objecti ves in "individualized" programs. Hov-;ever, it -is true, of 

m 

course, that such t^sts will have^differentiaf content validity for 

various specific prograrfis. The argument being made here is* that there 

shouldvnot be a bias for (or against) individualized vs. standardized 

programs in general. . . - * 

» * * *" 
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* Another araument used against standardized instruments for program 
comparisons is that sjnce standardized tests are designed to differentiate 

m 

among individuals, they aYe not effective in differentiating among 
programs. Such an argument seems to assume that the individual's score 
is the unit of analysis and that within-group variability attenuates the 
probability of obtaining signifi^cant between-group variability- However, 
this argument is not relevant if the unit of analysis is the classroom or 

- (■ ■ 

Selection of an Instrument for Assessing Achievement in Arithmetic 
The process of selecting, a math achievement test involved two general 
steps: (a) the initial .identification of potential instruments to be given 
further consideir^tion; and (b) the final selection of the instruii^.cnt te 
be u^Sed. 

In step (a), the following criteria iSere used in the instrument 
identification process: 

• The inst-r'Jinent *has ge>^e^al acceptance and wide use throughout 
the nation; 

• Strong evidence of content validity for^ currently used instruc- 
tional programs in math; 

• The availability of at least two parallel forms of the instrument; 

• The appropriateness of the instrument for use with second 
and third grade children; 

• 4 

e The time required for test administration is no more than 
45 minutes per subsection; 

• The procedures which are used in scoring the instrument are 
highly objective and efficient. 
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On the basis of these criteria, the following instruments were 
identified for further consideration: 

f Metropolitar> Achievement Test • 

• Bobbs-Merrill Arithmetic Achievement Test ^ 

• Comprehensive Test of Basic Skills - Arithmetic 

• Iowa Test of Basic Skills - Arithmetic 

• SRA Achievement Test - Arithmetic y 

• ^^Stan^ord Achievement Test - Arithmetic 

These initially identified instruments were then given a more in- . 
depth assessment, using information gleaned from tiests and manuals, dis- 
cussions Hith .putjjishers, technical reports, mental measurement yearbooks , 
and journal articles. In this assessment, the following factors v/ere given 
major consideration in the final selection process: 



• The existence of a sufficient range of achievement over 
which the instrument effectively discriminates to minimize 
the possibility of ceiling and/or flocrr effects; 

0 Adequacy of the instruments' general layout, required 
reading level and item tormats for use with second and 
third grade' children; 

• Adequacy of- interval consistency among items within each 
subsection of the test (where a KR-20 coefficient of .80 
is the minimum criterion);' 

• Evidence that various forms of th^TnHrument are parallel > 
(including: a minimum alternate foVrfi reliability of .85, 
comparable item diff icifl i ties between forms-r-^conjparable 
numbers of items related to specific objectives betvyeea 
forms); ^ ^ 

• The technical 'excellence in the procedures used i'n the 
generation of national norms; 

• Evidence that the instrument is culturally fair; 

• Evidcnce'that the time for conducting the logistics of 
scoring and relatect activities could be minimized 

(i .e. , less than 5 days ) . 
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After a critical review of the above instruments with respect to the the^ 
criteria, it was concluded that the 1973 Stanford Achievement Test (SAT) - 
Arithmetic should be used for both pre- and post-assessment' of achievement 
in arithmetic. The specific levels, forms, and test sections of the SAT 
to be administered to 2nd and 3rd grade students on 'pre-testi ng and post- 
testing are presented in Exhibit III-18. 

At should be noted that in the final review process, the SAT was 
not only found to be acceptable with respect to all of the criteria listed 
above, but also compared favorably wi^h the other instruments under con- 
sideration. The one exception to these findings dealt with alternate form 
re-li^^bil ity for which no information was available. However, otjaer 
evidence was present which supports ttie contention that the alternate foms 
are in fact parallel . 

- Some of the outstanding features of the SAT wbich provide a strong 
argument for its use include: ^ ' , • 

' A 

(a) The content of the SAT is compatible with the changes In 
curriculum and instruction which have taken place over 
the, last decade! Jhis outstanding asset is partially due 
to the recency of the development of the latest version of 
the SAT {i.e., 1973). 

(b) The SAT provides a rather complete coverage of the major 
instructional objectives found in the di verse 'math curricula • 

• * used today. This is partially made possible through the 
' " narrowing of the grade range which each level of the SAT 
addresses, thus making it possible to incorporate greater 
> breadth and uep'th for test objectives. 

'(c) The SAT provides "scaled ^scores" obtained through the use 

of Thurstone' s absolute seal ing . procedure. This has resulted 
in the development of a system of interbattery standard scores 
which permit the translation of ravA' scores' at each level 
. ; to absolute scores with comparability across test jev61s 

and forms within, each ^ectton of the' test. Scaled' scores - 
have the unique, adva^age of providing approximately equal 
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Exhibit III-18 



LEVELS, FORMS, AND TEST SECTIONS OF SAT - ARITHMETIC 
TO BE ADMINISTERED , 



Level 'and Form 


SECOND GRADERS 
Test Sections 


* THIRD GRADERS ' 
Level and Form Test Sections 


PRE-TESTING 


9 






Primary I 
Form A 


Math Concepts 
Math Computation 

Math Annlirflfinn 


Primary II 
Form A 


Math Concepts 
Math Computation 
Math Application 


POST-TEStING 


t 






Primai^y II 


MatJ^' Concepts 
Math Cogiputation 
Math Application 


Primary III 
.Form B 


Math Concepts 
Math Computation 
Math Application 



V 
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units on a continuous scale. Thus, these, scales are of 
vast importance in measuring achievement gains over ex^ 
tended periods or for use in groups of students which 
show extensive variability in achievement. The combined 
effect of points (b) and (c) fs tf)at a greater number of ' ' 
the test items are at an appropriate level for assessing 
students at a given grade without the loss of comparability 
of scores, 

(d) The national standardization of the SAT is outstanding for 
this kind of instrument. The standardization sample of 
275,000 students was selected to represent the national 
population in terms of geographic region, size of city, - 
socioeconomic status , ethnic background, and school owner- 
ship Oi.e., public vs. non-public) as v;ell as other factors. 

^In addition to the highly representative national nomis 
generated in the standardization process, there are also 
inner-city norms -a^vail able which may be more representative 
of the kind of population with which this study is involved.- 
Thus, both sets of norms may be used for making appropriate 
normative comparisons for each of the well -implemented in- 
structional approaches. 

(e) The SAT provides an index of instructional objectives which 
identifies and describes the behavior assessed by each item. 
Through the use of this system, it is possible to obtain 
n\ore meaningful comparisons betv^een local curriculum ob- 
jectives and the behavioral objectives called for in the 

.items. For this study, such a capability is important since 
it will provide a means of assuring comparable content 
validity for standardized and individualized approaches. 
This will be accomplished by identifying those objectives 
which are compatible with specific curriculum objectives; 
then by comparing the average number of , objectives v/hich 

, match this test for both the standardized and individualized 
programs. If they are not significantly different, it is 
reasonable to assume the test is program fair. ' If a 
program bias were foynd to exist for some of the objectives, 
than a separate analysis v/ould be carried but on those 
objectives which are common^ across the two instructional 
approaches. (See data discussion of Additional Procedures. 0 

(f) The items fncorporateij in the SAT'were car'efully constructed 
so as to be relatively free of bias with respect to race, 
religion, sex, and ethnic background, while at th^e same 
time' reflecting the cultural, diversity of children in t;he 



U.S. 




138 



III-98 



✓ 



Selection of an Instrument for 7\ssessing Achievement in Reading 
The procedure followed in selecting a reading achievement test was 
the same as that used in arithmetic in that the same two-step approach 
was taken, with comparable criteria used for initial identification and 
final, selection of instruments. Through this process, the following 
instruments iiere identified for further consideration: 

• California Achievement Test - Reading 

• Comprehensive Test of Basic Skills - Reading 

• Gates-MacGinitie Reading Test 

^ • Metropolitan Achievement Test - Reading 

• Stanford Achievement Test - Reading 

A critical review of the above instruments with respect to the 
criteria specified earlier resulted in no single instrument providing clear 
s^uperiority over all others. For this reason, the SAT was selected for 
use 3ince (a) it was defined to be of comparable quality to the best of the 
other instruments in this group with respect to the criteria used, (b) 
it has the same additional advantages that were mentioned for SAT-Arith- 
metic, and (c) by using the same instrument for arithmetic and reading, 
there are some additioral advantages which accrue, including com- 
parability of normative standards, and a simplication of the adrrji nistratibn 
and scoring process. The specific levels, forms, and test sections of 
the SAT to be administered to 2nd and 3rd grade Students on pre-testing 
and post-testing are presented in Exhibit III-19. In addition, by using 
the SAT for assessing growth in both areas, we anticipate greater efficiency 
and priority in scoring, a critical consideration in this study. 



Additional Procedure for Assessing Cognitive Outcome Measures 

One of the major testirtg concerns during deliberations among members 
of the project team focused upon the issue of program fairness / namely , 
whether or not the variance in content and sequencing among programs would 
be s6 great as to. preclude the use of an existing standardized norm-reference^! 
test aSu the sole measure for cognitive outcomes. While this, issu^ cannot 
be resolved until the specific programs to> be studied have been selected, 
and even though the SAT appears to have wide enough coverage to ensure * ^ 
content validity, we felt it-^propriate to describe the alternatives 'which 
the project team considered in arriving at a recommended additional 
approach. It should be noted that the procedures 'outl ined below and the 
recommended approach are vecy similar to that suggested ih^a study effort ^ 
^conducted by Dr. -StepTien Klein ( 1972). j ' ^ 

In our review of EPI£ and other critiques of individualized .prog>^ams , 
as well as analyses of appropriate materials inrVuded in inc;trurtinnal 
•packages sent by publishers, v;e found some variance among programs in the 
number of performance objectives and- proficiency levels far mastery, the 
degree of overlap of Objectives and skills to be taught, and the sequencing 
of content and objectives. In addition to these differences based on project, 
experience, . it was also agreed that the relative importance pf objectives 
varied and that the amount of time to achieve specific objectives would vary 
a'mong various types of studfents involved. In a similar manner, Klein has 
categohired the major factors any program comparison technique should take 
into account as follows: ' 

t success in mastering objectives; 

t relative importance of objectives and their overlap; 

t time spent in achieving objectives; 

' . iii-ioo 
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LEVELS, FORMS, AND TEST SECTIONS OF SAT - READING 
. TO BE'ADMI^^ISTERED 



Level and Form 



SECOND GRADERS ' IHlRt) GI^ADERS ' ' 

Test Sections Level and Form Test Sections 



PRE-TESTING 

Primary I 
Form A 



Vocabulary V 
Reading 

^Comprehension, I . 
Reading 

Comprehension II 
Word Study Skills 



Primary II 
Form A 



Vocabulary 
Reading 

Compnehension I 
'Reading 

Compreheosioi} II 
Word Study Skills 



POST-TESTING 

Primary II 
form B 



Vocabulary 
Readijig 

Comprehension I 
Reading 

Comprehension II ' 
Word Study Skills 



Primary III 
Form B 



Vocabulary ' 
Reading 

Comprehension 
Word Study Skills 
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e number and kinds of studnets involved; and 
9 cost and resources . 

Prior to reviewing Klein's work in this area, we considered various al- 
ternatives as described bglow. 

First, in addition to administering a standardized norm- referenced 
test, one could atso administer one of several criterion-referenced test 
banks (e.g., Zweig, Fountain Valley). We reviewed the following criterion 
testing programs: lOX, Fountain Valley, Prescriptive Learning, National 
Assessment Te^ts, and criterion testing programs embedded in the content 
of various programs. A critique of criterion tl&sting programs recently 
conducted by the Center for the Study of Evaluai:ion (Kosecoff, unpublished) 
was also very useful. Our general consensus after this review was that 
administration of any criterion te^t along with a standardized norm-referenced 
test would probably result in test "overkill" and lessen the degree of 
LEA cooperation; that none of the above tasts -would b'e suitable in their 
existing form and v/ould require some redesign (i.e., selection of i.tems); 
and that the cost of administering and scoring such, tests given the ' ' 
reporting date constraint would be prohibitive. 
_v • Second, we considered the alternative of either designing or 
developing a simple criterion testing program for the purpo'ses of this 
project. This alternative v/ould entail additional cost and would be ■ ^ 

extremely time consuming in addition to the above limitations or dis- 
advantages. In addition, we considered the alternative of utilizing 
the CohenVRandom House HILS computer-based criterion testing program. 
Essentially, this procedure would entail determining the specific materials 
used in the various programs and then requesting a computer printout on the 
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scores and/or performance objectives covered in the programs. This alterna- 
tive would require administration of criterion tests and/or program audits 
of individual student mastery records and verification of teacher accuracy 
and reliability in scoring. 

A third alternative considered was the-admi nistration of two or three 
nationally nomed standardized tests assigned randomly to students in the 
■ program/classroom; converting tes,t items to^stated performance objectives 
and/or skill areas; determining which performance objectives were common 

across program as well as those uniuue to specific programs; and then asses- 
ing programs in. terms of the number of '"common" objective's and unique ob- 
jectives mastered. The major advantages of this approach would be a lack 
of need for additional testing as in the case of the. administration of 
criterion tests, and lower administration, scoring, and logistical costs. 

The process of converting test items to performance objectives has 
uLL.i uvin- III cue Kuiaiiiu^uu ruuiic ociiOutb diiu eibewiiefc, Ddsya oti d re- 
view of the Kalamazoo testing system (e.g., a conversion of the Metropolitan- 
Achievement Test -- MAT) and discussions with representatives of Harcourt, 
Brace, and Jovanovich (publishers of the MAT); it was determined that such ' 
an approach was feasible although the number of items per objective would 
vary considerably. In addition, the conversion process would be extremely 

;tine consuming," although less so than develpping a new criterion testing 
program. The procedure for matching test items (now/ converted to performance 
objectives) to specific objectives in the program could either be done 
through reviews with individual teachers (e.g., as in the case of Kalamazoo) 
or -through a review of existing materials, relying upon project staff and/ 
or computer prescriptions from the HILS system 'mentioned earlier. Such a 
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materials review would be extremely time consuming. Dependence upon teachers 
could result in unreliable responses (e.g., a teacher might hesitate to 
' state she taught a specific performance objective if 'she felt students 
did not achieve that objective). Reviev/ by a panel of "experts" would 
probably be niore desirable for that reaspn, 

A review of 'Klein's publication subsequent to the considerations 
of alternatives by project team members indicated close similarity in 
identifying advantages and disadvantages. For example, while Klein indicated 
that standardized tests are developed without any intentional bias for one 
program over a-nother and would, hopefully, not favor the content of one 
program over another, he did argue that a standardized norm-referenced test 
may inadvertently favor one program over another in terms of emphasis of 
objectives measured; such tests might na*n:over all relevant objectives 
or even all those which overlap and would not consider the difference in the 
number of objectives cc^'ered. Siii^.ilarly , "curricjl uiii-c^bcddcd" tests v/culd 
not .allow for the handling of differences in test difficulties, number of 

* 

objectives covered, relative importance of objectives, or time spent to 
master objectives for cross-program comparisons. Assessing the techniques 
of developing criterion tests for objectives common across all programs, 
Klein noted that the major advantage here would be a program-free test 
measuring growth only on common objectives; however, it would not focus 
upon- objectives unique to each program; it vyould be highly dependent upon a 
larger degree of overlap between program objectives, and if overlap is 
slight, then only a small aspect of each program v/ould be considered. 
, , Moreover, this approach would not relate the importance of objectives 
across programs » 
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The concept of "program fair" testing (Popham, 1959) would measure 
programs on both comn^on objectives and idiosyncratic objectives of each program 

b*eing compared. While taking into account the most critical of thePsix 

/ 

factors described earlier, as Klein notes, this procedure has no provision 
for systematically handling differences in the relative importance of 
objectives, and the number of objectives the program is attempting to 
achieve. 

^^Jhe^^^conc^tual framework for analyzing alternative approaches as 
well as general findings described by Klein and those arrived at independently 
by members of the TURNKEY project team are very similar indeed. 
As Klein notes, "In addition, they (program-free and program fair 
testing concepts) both leave unanswered many questions which must be 
addressed, if we are going to perform realistic comparisons^ of different 
programs. For example, what are we to do if the objectives of the programs 
being compared do not overlr^p to a high decree, or if the^'-sh 
common objective? What if we are faced with a more complex situation such 
as when one program has a total of 25 pupils, takes all year, and ha5 10 ■ 
objectives; while a second program has 500 pupils for a seniester and- 
has only five objectiv-es, and only two of these overlap the objectives 
of the first program?" (p. 9) 

^ In the event that a prelijninary review of the programs selected for ^ 
study iio^icates the existence of these considerations, the application 
of this procedure bec^omes an even more important aspect of the study. The. 
proposed approach outlined below reflects a synthesis of the procedures 
recormiended by Klein and those independently derived by the project 
team. It should be emphasized that the major purpose of th^is set of 
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procedures is to determine the relative effectiveness of the various 

programs and not to measure the absolute amount of success of a specific 
program. * . . ^ 

1 . Select test 

Rather than selecting and/or developing criterion test systems 
fpr administration, we propose to convert the specific test items 
in the standard norm-referenced test (i.e., SAT) previously 
selected for administration to performance objectives. * The 
SAT will report scbres on^nndividual students as a criterion 
test does; moreover, it provides normative data-'Ce.g. , difficulty 
levels) on each item. To ensure broader coverage, it might 
be necessary to select an additional test with similar properties 
(e.g., the MAT), randomly assigning each test to students in 
each classroom. 

2. Determine relative importance of "objectives" 
** 

By using SAT, the process by wjiich the test was developed will 
ensure some general agreement "upon the relative importance of the 
skills being assessed. However, to ensure greater acceptance^ 
of the approach talcen and arrive at a higher degree of congruency, 
we proposed to have a person extremely knowledgeable about the 
content of each of the selected programs (e.g., the LEA 
curriculum staff ) assign weights on a scale of 0 to 5 to each 
objective in ascending order of inipot tdnce, or dsbiyn a cold! 
of 100 points to all objectives, with limits on the maximum 
number of points to, be assigned to any objective. As an al- 
ternative to using firm and LEA representations, a panel of 
disinterested, objective expert.s could be assembled ii^e Kleint 
p. 11). / 

3. Determine the degree of success achieved by the program 

As Klein correctly notes, several problems exist in measuring ' 
success if different objective or norm-referenced tests are 
used: (a) the need to convert scores to a common scale since 
the number of items and methods of assessment will \;ary; (b) 
difficulty levels vary among objectives and instruments, 
as do "equal intervals" among items; (c) variance in instrument, 
reliability, and validity. By using the SAT, however, the 
seriousness of these problems diminishes considerably, 
especially in light of the fact that normative data on item 
difficulty can be determined. 
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The proposed steps for determining program "success" under this 
option would be to: ^ 

• Determine the number of students mastering *each objective on the 
SAT who did not exhibit "mistery" (i.e*/', "get it right") on the 
pre-test. 

^ fi? ^ ' 

• Calculate the percentage of students who had not previously 
mastered any given item mastering the >vt^em on the post-test. 

• Multiply this percentage for each 'objecti'v&stjy the assigned 
(v/eighted) relative importance, 

• Multiply this product by a "difficulty factor" *for that 
objective obtained by subtracting the 'proportion of appropriate 
students (i.e., second grade, ninth* month, etc.) in ttie* 
national norm sample v/ho responded correctly to that item 
from the number 1.0. • * . 

• Sum up these products over all objectives covered by the tesT> 

♦ 

Exhibit III-20 illustrates this procedure for a simplified application.^ 
To summarize, the advantages of this program- fair approach are: 

• A reliance on standardized norm-refernced test which v/ould 

. capitalize on the curriculum studies -use^d in the developmeat 
of such tests ^ the national norm both^for large subtests and ; 
tor individual iteos., the judgr^ents niade^by- panels of national- 
experts on the rahg'e and scope of the i terns/object tves to 
include, as well as on all other test ^ispects sych as re1ial)ilit 
validity, scoring eSse, etc.; 

0 A re/l^ection of the local importance assigned to the, teaching 
of particular items in a way that for a^T-other aspects ;being 
equal rev?ards the program which pursues* more objectives than 
one which pursues less. * * ^ 

9. A reflect:ion of the difficulty of the objectives pursued by 
^ any gi ven p>;pgram. . ' ' ^ 

Tfie above approach differs from that proposed by Klein in several 

) 

respects for reasons unique to this study. First, giv^n his formula 
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Exhibit III-20 
Page 1 of 3 



SIMPLE EXAMPLE APPLICATION OF THE ADDITIONAL PROGRAM COMPARISON SYSTEM 
% 

PROGRAMS: I and II 

TEST: ' One yhich has 4^,1 terns; each item taken to be an objective 
.STUDENTS: Ten students sampled for each program 



RESULTS: PROGRAM I 

Objective (Item) Number 
Student T 2 3 4 



A X * * 

B ' * X . * 

C * X 

D ' X * X * 

E * X * 

p * * * 

G XX 

H * * X • 

I * * * 

0 * X * ■ . 

(Key: X indicates "mastery" shown on pre-test; * indicates' "mastery 
of item on post-test not previously mastered; blank indicates 
no masterly shown) 



PROGRAM II 

Objective (Item) Number 
Studeiit 12 3 4 

A 

g * * * * 

C ,X * 

D * * * * 

E , * . * 

p * * * * ■ ' 

, G * * * * 

H ■ . ' * * 

I * * 

J . X * 

(Key: X indicates "mastery" shown on pre-test; * indicates "mastery" 
of item in post-test not previously mastered; blank indicates 
no mastery shown)- 
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Exhibit 1 11 -20- 
Page 2 of 3 



SUMMARY OF RESULTS: 



Program I- . ~ Program II 
Objective Number Objective Number 

1 2 3 4 12 3 4 

■* • ■ ■ "" " o 

Number students not showiYig 

•previous "mastery" 7 7 7 , 9 • 9 10 9 10 

Number now showing "mastery" 6 6 3 4 3 77 8 

Ratio of latter to former .86 .86 A3 .44 .33 .70 .78 .80> 

.Average Percent "Mastery" in 
all Objectives ' .65 .65 ^ ^ 

Based on summary, above programs are equally "effective". ^ 

However, considering our other factors: ^ 

Relative Importance of Items in Each Progra m (in' both cases the sum of 
the weightings over all objectives equals 1.0) 

Program I " , Prograri II 

^ Objective Number Qbjectiv.e Number 

1 2 3 4 1 2 3 4 

- 

Relative Importance .25 .25 .25 .25 .10 '.20 .20; .50 

Relative Difficulty of Each Item Relative to National Norm 

Proportion of National 
Objective Number • Sample Showing "Mastery" "Difficulty Factor" 

1- .10 - - '-90 

.2 . .50 .50 

- - 3 • . .50 ■ . .50 

- 4 ^ .90 • ■ -10. 
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Exhibit III-20 
Page 3 of 3 



^ Calculation of Progranj-Fair Effectiveness Results 



^ Program I ^ Program II - 

Objective Number ' - Object i\re Number 



1 




2 


3 


4 


1 


2 ' 


3 


4. 


Percentage of students who had not 
previously mastered the item 
mastering the item on post-test . 


85 


.86 


.43 


.44 


.33 


.70. 


' .78 


.80 


Relative Importance 


25' 


.25 


.25 


.25, 


.10 


'.20 


.2a 


.50 


Difficulty Factor 


90 


.30 


.50 


.10 


.90 


.50 


..50 




Product of above 3 values 


19 


.11 


.05 


.01 


.03 


.07 


.08' 


.04 


SumfCver all objectives on 
program fair effectiveness 
measi^es 






.36 








.22 





Jhus, 6n a program-fair _basis, Program I appears to be the more effective program. 
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Where S fndi cates success on the object! vesin terms of to^al weighted' . ' 
spores, N indicates the number of pupils in the. program „"T indicates amount 
of pupil time in the program, C indicates total program cost, and E 
indicates prograrp effectiveness — we propose to delete "C/N" (program 
cost per pupil) for c-omparison purposes here. Rather, the cost factor will 
be-4nc4ttded--iT.-^c uob l^^ f ftL L i ^en-ess- arraly^s 0"r*^"a&soTLrEe^ rather "than " 
"relative" program outcomes and treated 'in more detail than it is . 
treated by Klein (who acknowledges the somewhat superficial treatment of 
this factor in his formula) and.vnth a different priority focus (e.g., 

♦ 

.comparing "mainstream" vs. "pull-out"). Second, T (tiip.e in program) has al 

been deleted since:' (a) differences in student time in the program will 

be relatively similar due to the sample design; '(b) where time difference? 

occur, they will be reflected in the separate cost-effectiveness analysis 

'since i;nost resources consumed by a student ar-e time-related. These factor 

deletions make clear the implication that program effectiveness is 

Synonymous' With success on_ the weighted objectives. 

It should be noted that the procedure outlined here could be carried 

out in a number of different configurations: 

Q Comparisons of the number and weightings assigned to objectives 
■ by, program; 

0 Compar^isons of- programs without weightings across objectives 
(approximating the standard scoring routine except for con- 
sideration of item difficuJty). 

Selection of an Instrument for Assessing Students ' Solf-Concep t 
•As in the earlier mentioned selection procedures, a fwo-st^p process 
was. us_ei for^ selecting a. finaljnjtruraent^ The cxiteria^ thatwere used in 

» 



in identifying poter^tial instruments for inclusion were: 

• Age appropriateness of tfre instrument for use with 2nd and 3rd 
grade students (including: nature of instructions and test* 
presentatioHf level of item vocabulary, and time require- 
ments for test administration - maximum limit of 30 minutes 
v/as used); 

* • 

• The procedures which are used in scoring the instrument are 
I high^ objective; 

• The existence of one or more subscales related to "schopl 
self-concept." 

, The instruments which were identified for further consideration on the 
basis of the criteria were: 

t Self-Concept of Ability Scale (SCAS) 

• The Piers-Harris Children's Self-Concept Scale (CSOS) 
(revised version for grades l'-3 for^USOE Longitudinal 
Survey, 1973) 

The major reason that a larger set nf instruments were not identified was 
that most instruments which assess self-concept are not age appropriate 
for 2nd and 3rd grade students. 

A, more indepth comparison of these two instruments was thenmade> 
giving major consideration to, the following factors: 



• Construct validity of the instrument giving special^ con- ^ 
sideration to such things as correlational and factor 
analytic findings that are conipatible with theories of 
self-concepi: (Wylie, 1961); 

• Evidence of .correlation with academic achievement; 

• * Evidence of internal consistency among items on each scale;' ' 

6 Evidence of (short-term) test score stability; 

0 Evidence of high correlation with other instrument's whtch 
assess self-concepts 

The comparison conducted between the.SGAS and <hfe CSCS resulted in 

the CSCS b^ing found to be far superior. The CSCS was also found Jo be 

acceptable with" respect to 'all of 'th6 factor's given niajor cons-ideratfoh, 
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and thus, will be incorporated in the study for asl|e^^44)^ self-concept. 



(It should be noted- that the Purdue Education Research' Center recommended 
the use of this instrument in their proposal for this study based upon the 
validation results from its field testing in the Indianapolis Put^Tig^ 
Schools.) However, in order for this instrument to be appropriate for use 
with 2nd grade students, the item must be read by. an examiner: Thus, an 
oral presentation v/fll be used for both 2nd and 3rd grade students with 
the scores obtained on all factors (on the revised form): feeling self, 
school self, and behaving self being considere'd separately. 

Because the Piers-Harris CSCS instrument, particularly the -revteed 
form for grades 1-3, might not be widely known, a brief description, of the» 
development process and assessment of correlation with other variables is 
appropriate^ as excerpted from McDani^l , et. al . (1973). . » \ 

Development of In strument for Grades -1- 3 ^ 

a shortened form of the P iers-Har ris Children'^ Self-Concept.Scale 
was developed^. To select items for the s'hort form,'" data from two 
earlier studies conducted at Purdue* were used. In, the first study, 
the full Piers-Harris scale was administered to 413 3rd graders 
^attending predominately black ghetto schools in Indianapolis, In 
the second study, .the Pier^s-Harris was administered to 119 2nd grade 
white suburban students. Item analyses were computed for each .sub- 
scale reported by Stanwyck (1972) ajid for t-bose subscales repcrted by** 
Piers-Harris (1959). Items ini tially ^chosen for the shortened scale 
v/ere those which correlated at ]east .30 with th^e total and appro- 
^ ^ priate subscale totals for the two studies. These items were inspected 
for appropriatehess of vocabulary for 1st grade children. An attempt 
also was made to balance the number of items 'representing each of the 
^factors.' The final form coat^ained 40 items which coOld be 'grouped. 
, into three of the factors reported by Stanwyck: feeling self, 16 
% . items; school self, 14 items; and behaving self, 10 items. 

' Answer sheets with Yes-No response positions were prepared for primary 
gratte children. Small pictures (e'.g., a dog, fish, boot) were used as 
, place markerb for. the items. 

' • - 
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This test v/as- administered to V68 1st grade pupils in Indian^ipolis. 
, Out of this sample, 25 answer sheets were discarded because of a 
larger number of omissions or obv>61^5^ irregularities in responding. 
The following analysis, therefore, is based on the responses of 143 
pupils. (The Item analysi'S is summarized in Exhibit III-21.) 



Item-subscal Acorpelations are considered more important than item- 
total score correlations in view of the effort to obtain independent 
scales. 

• ^ 

Four of the 16 items -on subscale 1 fall below'the criterion of .30 
correlation with the subscale score. One item in 'each of the other 
subsoales failed to meet this criterion. The* wording of each of 
these items, logether with thirteen items identified by examiners 
as containijig troublesome vocabulary^ v/ere revised before inclusion 
in the final form . . . The results reported in this section are 
for the'wording used in the Indianapolis testing. The reliability 
of the subscales appears to be tod low to warrant analysis of -indi- 
vidual scores, (p. 48) However, they are of sufficient magnitude 
for use in group comparisons. 

Correlations with Other Variables (Grade 1) 

The total self-concept score for Ist'^rade pupils was significantly 
correlated with achievement scores and some measures of independence 
and affiliative behavior. It was not correlated with -selected 
measures obtained fron peer ratings or parent questionnaires. (Exhibit 
III-2-1 presents the correlations betv/een self-concept total score 



The total self-concept scale, score was correlated (r = .23, p .01) 
with* reading achievement and (r = .42, p .01) with math achievement 
as measured by the Stanford Achievement Test. A significant correla- 
tion was found between the self-concept score and the independence ' 
subscale^ (r =\45, p .05) and the sociaV subscale (r = .59^ p. .01) 
of the Pupil .Information Bookl-et (Mcbaniel, et al, 1973). No signi ficant 
correlations were found between self-cqncept and the subscales of 
the Peer Ratings. Self-concept v^as not correla^d with the Maturity 
Demand Subscale of the Parent ^Questionnaire (Hcraniel, et al , 1973) 
(F). 48) , ^ ' _ ■ 

It is felt by^the TURNKEY st.udy team that significant correlation between 

the total self-concept scale an*d reading/math achievement is in no v;ay a 



existence of these significant correlation provides evidence of construct 





V 



drawback to the use of the instrument as intended in the study design.' The 
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Exhibit III-21 
Item-Subscale and Item-Total Correlations (Grade 1)* 
Piers-Harris Children's Self-Concept Scale Adapted for Primary Grades ' 



Item - Subscale Correlations 



L UciU 


T 
1 


T T 


T T T 

111 


1 0 td 1 




Poo 1 1 nn 
X ICC 1 1 iiy 




Dcna V 1 ng 


Score 


1 


.30 






33 


2 ' 


.32 




- 


29 


3 


.34 






29 


4 


.34 






18 


5 






45 


23 


•6 




.32 




24 


7 




.41 




22 


8 




.44 




^21 


9 


. 29 






22 


10 




.53 




37 


11 ' 






. 54 


40 


12 






43 


29 


13 




" 42 




17 


14 




. 55 




32 


15 
1 %j 




27 




^^7 


16 






14 ' 
. 1 ^ 


27 


17 






.55 




18 - 


^ 18 






'07 


19 


36 






33 


20 




47 


V 


• ou 


21 

£- 1 




63 


• *t 1 


22 






51 


46 


23 




.47 




26 


24 




54 




35 


25 ' 










26 




32 






27 






■? 




28 » 


.45 




32 


29 


. 39 






3? 


,30 


.41 






.22 


31 


.27 




• 


.16 


32 


■.29 






.17 


33 


.30 






.29 


34 






.52 


.39 


oD 


/I Q 
. HO 






.id 


36 




.42 . 




.29- . 


37 






.37 


.28 


38 


.33 






.28 


39- 






.43 


.31 


40 


.47 






.52 • 


N = 143 










Mean% 


7.64 


8.81 


6.40 


22.85 


SD ^■ 


2.72 


2.67 


2.32 


5.64 


■KR-20 ■ 


.48' 


.67 


.53 


.73 


*Reproduced frpm McDaniel , 


et al . , 


1973, p. 49. 
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validity for the self-concept instrument, at least regarding student 

performance^ in school, Kov/ever, since the amount of variation in reading 

achievement scores explained by variations in the total self-concept scale 

Xr = .05; 5% explained variation) and the. amount of variation in math 

athievei^t^m: scores explained by variations in the same Scale (r^ = .18; 

18% e^lained variation) are both relatively low, it is. not reasonable / 

to conclude that achievement tests are already measuring whatever the 

total self-concept scale is measuring^ Thus, the correlations shown in 

Exhibit III-22 are considered appropriate. 

Selec ti on of an Instrument for Assessing Student's Attitude T oward 
School 

Again using comparable criteria for initial selection as was used 
for the student self-concept instrument, the following instruments were 
identified: 

0 Attitude Toward School (ATS) 

0 My Class Tnventory (MCI) „ •, 

0 School Attitude Test: Oral Form (SAT: OF) ^'^^ 

0 School Sentiment Index (SSI) 

These instruments were, compared with respect to their desirability - 

for use giving major consideration to factors comparable to those used^ 

previously for final affective instru4nent selection. This comparison 

resulted in the selection of the ATS for use in this study because- it 

was found' to be acceptable with, respect to al'l major factors with the 

exception'^of ;'high correlation with other instruments which^assess students' 

attitude toward school" for which no evidence v/as avail-able. In addition, 

the ATS was found to be superior to the other instruments considered. 

Scores obtained on all subscales: attitude toward school work, attitude 

toward teachers, and attitude toward school in general, will be considered 

> • 
as separate variables. ^ i r ^ 

• lob ' . . 
IIM16 
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Exhibit III-22 



Correlations between Self-Concept and Selected Variable? (Grade 1)*** 



Scale N Correlation 



Reading Achievement 138 .23** 

Mathematics Achievement 137 .42** 

Peer Rating Scale 

Individual Action 138 . .04 

Affective Relationships 138 .13 

Pupil Information Booklet ^ 

Independence 16 .45*" 

Social 16 .59** 

Parent Questionnaire 

Maturity Demand ^ 63 -.06 



p .Uw» v^iic tail ttibt; 

** p .01 (one tail test) 

*** Reproduced from McDaniel , et al, 1973, p. 50 
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The item analysis for the Iidianapolis data revealed no item- 
subscale correlations below .30, with the vast majority being 
above .40. The alpha estimates of reliability for this testing 
were 75 for attitude toward schoolwork, .79 for attitude toward 
teachers, and .81 for attitude toward SiilweM" general. The 
reliability for the total scale was/^1. (The Percent of students 
choosing each response for^Fortn V c/n be found ifn Exhibit Hl-^^. 
The item-subscale and item-total p6rrelations arN^ presented in 
Exhibit 1 11-24.) 

these 'data suggest that Form v/of the attitude toward school 
measure Is sufficiently well developed for use in subsequent 
studies. There is, however, i substantial intercorrel?tion 
among the 'sutiscales (schoolv/ork and teachers, .61; schoolwork and 
school in g^eral , .76; and tea'cher and school in general, .69). 

The high correlations suggest that the present instrument i/ 
essentially unidimensional in nature and may be measuring a 
single generalized orientation to the school environment. Whether 
this is an artifact of the instrument or reflects the actual nature 
of school attitudes cannot be determined at this time. Despite the 
apparent high overlap between the subscales of this instrument, 
there is son^e evidence from the correlations which follow (Exhibit 
III-25) that the subscales represent a degree of orthogonality. 



Correlatio ns with Other Variables (Grade 4) - 

It, would seem loaical that a child's attitude toward school would 
be related to several other variables. First, his general aoility 
arid achiever.ont scores should be important, assuming that school 
attitude is not likely to be high if school is too difficult for 
the student, or if he is not doing we,:il in the work required. Second, 
if a child's self-concept is markedly low or high, one might expect 
some generalization and projection of these perceptions onto the 
outer environment. Finally, it would seem that the parental aspira- 
tions for the child-and the parental acceptance of the child would 
have a bearing o.n the way the child relates to both the schoolwork 
and the school teacher. ^ 

Lo^•/ but positive correlations were found between attitude toward 
school and I.Q. measured here by Raven's Coloured Progressive Matrices 
(Exhibit II 1-25). . . Orthogonality of the subscale is suggested 
by' noting that the correlation between the Raven's and" attitude toward 
sch oolwork .is higher than the correlations between the Raven s and the 
other two subscales of the attitude 'tests. 

4 ' 

The highest correlations were found between attitude toward school and 
measures of achievement. These ranged from .28 to .43 and all were 
significant beyond the .001 level. Here again, correlations between 
the subscale attitude toward schoolwork and achievement were higher 
than those for the other subscales. These correlations are among 
the highesc found in the literature. . . 
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Exhibit III-23 



Percent of Students Choosing Each Response (Grade 4)* 
Attitude Toward Schf)Ol 



Number 


A 


B 


C 


D 


£ 


1 


31 


6 


15 • 


16 


■ 32 


2 


17 


7 


8 


19 


49 


3 


14 


6 


19 


11 


49 


4 


28 


5 


7 


12 


48 


A 


-15 


3... 


^ 12_ 


13 





6 


30 


7 


22 


7 


32 


7 


24 


8 


22 / } 


17 


28 


8 


22 


9 


14 ^ ^ 


19 


~ 36 


9 ■ 


14 


7 


14 . 


13 


51 


0 


12 


8 


. 16 


18 


46 


1 


13 


6 


33 


8 


38 


2 


31 


6 


8 


9 


42 


3 


27 


13 


18 


14 


28 


4 


36 


11 


10 


13 


29 


15 


24 


9 


•n 


14 


41 


16 


20 


8 


10 


14 


47 


17 


28 


9 


15 


14 


33 


18 


53 


8 


5 


7 


2f5 


19 


13 


5 


8 


20 


54 


20 


17 


3 


8 


rs 


56 


21 


25 


5 


22 


9 ' 


38 


22 


44 


8 


8 


^ '9 


32 


23 


11 


6' ' 


16 


26 


41 


24 


18 


6 


13 


13 


48 


25 


20 


6 


12 


14 


47 


26 


52' — 


17 


.9 


6 


" "15 


27 


23 


7 


23 


12 


34 


28 


9 


5 


6 


19 


60 


i9 


26 


6 


14 


13 


39 


30 


14 


9 


- 22 


16 


38 


31 


22 


8 


21. 


16 


33 


32 


27 


9 


17 


14 


30 


33 


24 


17 


14 


12 


33 


34 


26 


8 


14 


11 


40 


35 


10 


3 


13 


21 


52 


36 ' 


27 


12 


14' 


7 


39 


37 


32 


8 


18 


13 


28 


38 


40' 


9 


12 


8 


29 


39 


7 


4 


9 


12 


66 


40 


29 


7 


28. 


4- 


29 


41 


21 


11 


9 


6 


53 


42 


21 


7 


12 


13 


47 


43 


23 


8 


13 


13 


42 


44 


33 


10 


14 


7 


33 


45 


12 


2 


12 


T3 


59 



.uced from McDaniel , et al , 1 973 , p. 38 
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Exhibit III-24 






Item-Subscale and Item-Total Correlations (Grade 


4)* 






Attitude Toward School 












ocnoo 1 1 n 




* 


Item Number 


ocnoo i wor K 


1 eacner 


oenera 1 


Total 




1 . • 


• 




.50 


.53 » 




2 


.47 






.41 


r 


r 3 




.60 




.54 


V 








.49 


.45 ' ■ 




5 


.53 






.45 




" f 




.55- - 








7 


__ 




- - - - _ — 

.53 


.trjr 

.44 




8 






.51 


.47 




. 9 


■ .51 






- .45 




10 


.34 






.30 




n 




.37 




.28 




' 12 




.57 




.53 




13 


.52 




.62 


.58 




14 








.45 • 




15 




.65 




.61 ^ 


• 


16 






.66 


.52 




17 




.38 




.. .28 




18 








.52 




19 








.58 




20 




.67 




.59 




21 




.45 


• 


.39 


• 


2? 






.48 






23 






.36 


.42 




2A 


.42 






.31 




25 - 






.53 


.54 




ci) 






_28^ 


^ .30 




'27 


.63 






-.63 




28 




• .46 


• 






' 29 






.47 


.46 ■' 




30 




• .43 




.29 




31 


• 


.61 




.52 




32 


.42 






.32 




33 






• .58 


.52 




34 


.54' 






.46 r 




35 


.49 






, .42 




' 36 




.57 


• 


.48 




37 




• 


.55 


.54 




38 


' .59 






.55 




39 




.40 


A 1 t 

.41 


.42 




40 






.38 




41 


. .51 






? .42 




42 




• 


.45 


■.41 




43 , 


.30 






.24 




44 






■ .39 . 


.35- 




--—45 




. 51 




.36 




N = 179 












Mean 


49.2 


f 48.5 


54.8 


152.5 




SD 


10.7 


11.4 


13.8 


31.9 




Alpha Coefficient .75 


.79 


.81 


.91 


o 


*Reproduced from McDaniel , et al 


., 1973, p. 


39 
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Exhibit III-25 



Correlations Between Mtitude Toward School 
and Sel'ected Variables (Grade 4)^ 



Variable 


N 


Schoolwork 


Teachers 


School in 
General 


■ Total 


Raven's Total Score 


169 


.24* 


.15 


.15 


.20* 


Reading Achievement 


179 


..40** 


■ .^J5** 


.34** 


.40** 


Math Achievement 


179 ■ 


.43** 


. 28** 


.32** 


.38** 


Self-Concept 


175 ' 


.39** • , 


.24** 


.29** 


.34** 


Parental Aspiration for Child 


58 


/ -.32 


_ _ 3"** 


-.32 


-.37* 


Parental Acceptance of Child 


86 


.24 


.24 


' .24 


.27 



* Significant at .01 level 
**Significant at .001 level 



1 



reproduced from McDani el et al., 1973, p. 40, 
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Below, a description of the development process and correlation with 
other variables, excerpted from HcDaniel , et al (1973) is presented. 
DevelWient of Instru'iient . 



' f?vrLccess?vTJn' °i n^' ^""^^^""^ ^° '""^^^^^ ^"^'^ude toward school, 
tive successive forms of a questionnaire were designed and tested Each 
form was tried m a school setting in order to (1 ) test the Uens (2) 

me ho s'anS'rlf ''''' stitfstical a d anal) c 
SiJecs^ tesSiigl approaches available only through^ 

s^?es^^or'a{??t?;H!?'°"H ^'t ' '15-i temTnstru.ent with sub- 

y?idf rcht-^?^^ 

4th grdde, ,9 in the 5 grace, and 23 in the 6th grade. 

Teacheifs in grades 4-6 were asked to rate their students on attitude 
toward school The 5th and 6th grade teachers rated their pupils on 
a scale rroni 1-100.^ The 4th grade teacher ranked her pupils to indi- 
cate the iiiObL positive to least positive attitude toward school. The 
correlations between the total score on Form IV and teacher ratings 
were .69 for grade 4, .60 for grade 5, and .66 for grade 5. 

Fof<f»-day test-retest correlations for Form IV are presented below: 





--Grades 




J_ 


5 


6 


.74' 


,86 


.76 


.55 


.74 


.68 


.83 


.93 


.76 


.86 


.91 


.80 



Schoolwork 
Teachers 

School jn Gene,ral 
Total 

The correlations between teacher rating and total score on Form IV 
are qu.ite high when compared" wi th other studies in which teacher 
ratings were compared to other criteria . . . Scores- on Form IV 
apparently measure attitudes which can be observed by teachers. 
The high test-retest correlations suggest score stability over a 
short period of time for the total score. 

Form V, the final form, represents a further effort to 'improve 
subscale reliability and internal consistency. Thirty-nine of 
the '45 items used on Form V had shown substantial item-subscale 
correlations in the prev.ious pilot studies. Six new items wSre 
' developed. ^ _ . - - 

The three subscales of Form V are attitude tcv;ard schoolwork, 
f 14 Items; attitude tows rd tjeachers, 14 Items; and attUuTe 

toward school in general, 17 items, form V was administered 
» to 180 students in seven 4th grade classes in Indianapolis. 

^ • 
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JmJ^d'^^honr'"2 ^1^°^- 'PP-^"^ relationship between 'attUudf) . 
^^^^^1^12^ relationship, between se1. perceptions , 

tiv'Ifi;''re?at^d'' P^^^^^al aspiration for child were nega- 

tively 1 elated. Correlations ranged from -.37 to -.32 Items con- 

? e^ "r'e?e^tr?rV"^ °" ty.^■^^r.nt Questionnaire '?hese , 
Items refer to the degree of concern th'e parents have for the success 

of their childrtn. The negative correlations between parental achievement 
motivation and attitude toward .school may be related to Erhlich's observation 
that high enthusiasm for elementary school seems to l?e associated with low 
socioeconomic status. Perhaps parents with a high level of concern tend 
to expect too much of their children which could result in the child's lower 
attitude toward school. This is an area relatively untouched by research 
and one that needs further investigation. 

Positive correlations were found between attitude toward school and parental 
acceptance of child. This could be somewhat anticipated. Although none 
of these correlations were significant at the .01 level, all" were" significant 
at the .05 level . ^ - 

Correlations were found between attitude toward school (Form V) and several 
other educational and psychological variables designated for the lonoitudinal 
study. vThese nre nrp^pnted in Lxh^bit I!!-25.) # 



|flg£110D-0.1jJLj^strum8nt for MeasjjringTpacher'^ Assessment of 
Classroom Environifient ~ 

For this area, the set of criteria used in the initial identification 
' phase were as follows: . / 

9 The procedures which are used in scoring the instrument are highly 
objective; ^ 

e Appropriateness for use in the assessment of grade school class- 
rooms; --^ 

0 Acceptable internal concsistency among items within each scale 
to be used where a KR-20 coefficient of .50 is the minimum 
criterion. 

The instruments which were identified for further consideration on the 
basis of the above criteria v/ere: ' ^ 

0 Class Activities Questionnaire (CAQ) 

• - ■ 1G3 
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• Learning Environment Inventory (LEI) 
f Minnesota Teacher Attitude Inventory (MTAI) 
f 



Organizational Cliraate Description Questionnaire (OLDQ)-^ ~ 
f Organizational Climate Index (OCI) 

t Purdue Teacher Opinionaire (PTO) ' ^ 

c 

% Social Climate Scale (SCS) . 

A more indepth comparison of these instruments was then>made, gVving, 
major consideration to the following factors:/ ^ i - 

f Construct validity of the 'instrument giving special consideration to 
such things as correl atiomal and factor analytic findirrgs that 
are conipatibl^e'wi th theory incorporating classroom environment 
as a construct; • , ^ ^ , , ; 

' ' ' '* ' . * . ' 

• Evidence of correl atibn 'with academic achievement; 

t Availability of elementary school norms. ' \« 

A critical review of the above instruments resulted in the^ selection of 
the OCI for use in this study. The* one particular impressive- characteristic ' 
ol this Jus triiraent was the strong positive evidence available regarding, ' 
its construct validi.ty and its demonstrated "power"" in assessing individualized 
programs. For example, Kelly, et ah (1^73), in their study of teachers" 
perceptions' of school climate resulting from the' implementation of IGE * 
and other studies sponsored by yi'/D/E/A/ Kettering, found the OCI instru- 
ment to be fairly consistent in reporting the following findings for inner^' 
city schools: . - , • • ' . 

Increased iiopJementation of the IGE model in inner-city- school s sfiould: 
(1) lead to an improvement in the I ntel lec'tual CI imate by increasing ■ 
teacher, interest in-intellectual activities, social action, and 
improvement of personal and interpersonal effecti venes's ; (2) lea'd 
to teacher behaviers which reflected hard work, and preseverance ex- 

pres-sed in an increased committment of the needs of the sphool ;' 

(3) lead to increased sharing of help and expertise; (4) result 
in greater respect for each other's integrity and yet foster a 
willingness -to help one another; (5) hel p to develop uniproved 
organization and .procedural thoroughiless in the planning and • • 

. ■ ■ , , . .. , ' , ' , _.•„■ 
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conducting »o,f duties aimed at accomplishing^th-e purposes " 
of the school; and (6) create- greater ope'nness and flexibility 
in^the -day-to-day operation of the building. , . - " * • 

Scores obtained on all factors cprr.prising OCI -- intellectual cl 
achievement standards, practicality, supporti veness, orderliness, and 
Impulse control -- will be considered as separate variables. Finally, 
-school clirates isolated through combined factor ana-lyses Of the OCI 
and the Stern Activities Inde)^ were found to differentiate between 
grade levels and to be related to other measures such as teacher turn- 
over. ■ .• ' 

The abo-ve validating information does^ suggest t?hat the OCI is 
capiable of distinguishing between institutional environments in ways 
.that might be expected, given olher information o'n the organizations ■ 
studied, . -. 
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TASK 5 -- DEVELOP ANALYSIS PLAN ' ' ' • 

Throughout the previous, four tasks the design aspec'ts of the proposed 

study have been developed'. In Task 1 we- described the y-ariables to be 

* * I* 

studied and how the data for each would be obtained In 'Task 2 v/e ■ ^ 

described the sampling plan, and in doing so we presented a number of 

critical analysis issues including: ' ' ' 

t The Unit of Analysis a "classroom" see Exhibits III-7 
and III-8. ' ' 

• Th^ Basic Design Framev/ork centered in the four dimensions 
of our definition of individualization, mainstreaRi vs, pull- 
out, and reading vs. math, see Exhijbit 1 1 1-9. . - • 

6 Why the Proposed Study is to be. Centered on Grades 2' and 3 » 
the continuous nature of this grouping being consistent with 
the concept of individualization, the availabil ity of 
appropriate instruments at these grade levels, and ,the- 
somewhat similar nature of programs at these two grade levels. 

* In this task we wi r I first discuss:.some of the general issues related 
to our analysis plan, issues that are related to the aspects of the design 
listed above. We will then djscuss the specific research questions to be 
addressed by the proposed study. And finally, we will describe additional 
analyses that will be^ conducted teyond the basic statistical comparisons 
discussed to that point ~- including cost-effectiveness analysis simitar 
to the work done by TURflKEY in the development of tfie Michigan Cost- 
Effectiveness Model for Compensatory.- Education Programs.. 

Befo.re discussin^g the specific research questions to be addressed,' 
a number' of design features should be discussed further. The first of 
^ these, features to be explored beyond the discussion presented under 
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previous tasks is the grade 2 and, 3 focus the proposed study. It 
was mentioned in Task 2 discussion that if the gradies studied were not 
contiguous, compar^vspns between individualized, and standardized programs 

T 

at the saire grade levels might ^be biased against standardized programs 
because of 'differences in material availability, rather than trUe differ- 
.ences in the effectiveness of one instructional approaches, the other. 
From an analytical point of view, it is also .suggested that .for the 
purposes of analysis'^11 results should be discussed 'and studied in terms 
of grades 2 and 3 combined, both for the individual i zed. programs as well ^ 

results at these two contiguous levels, this bias problem is minimized* 
Separate analyses. by grade leve.l/-could always be generated if a conparison - 
of results by a category such^ as the nominal grade l^vel assoctatedM.'i th 
a given program for -administrative purposes is considered desirable. The 
dds-ir.e to u-se cpp.bined results for these grade levels in our analyses can 
be readily accoinaiod^a ted- wi thin our sampling plan as already indicated by 
pla-cing a constraint on 'the- pveral 1 plan of having to include^out -fhe s^rne 
^ number of nominally second grade classrooms and nominally 'third grade 
classrooms \x\ each sampling cell. - ' . . 

* The next design feature to be addressed here is the nature of the 
basic design framework discussed previausly in Task, 1 and in more detail 
in Task *2. To the casuctl reader it may seem that th^a Definiti ona,l Matrix 
(sefe Exhibit III-9) relied up&njn thi? baU c' f ramev/ork differs' in concept, 
and approach f rpfir the steps outlined in our original Proposal for developing 
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operational definition of an individualized br a; standardized pro.grani. 

£71 our Proposal wfe suggested that we assign rati'ngs to the characteristics 

that would make up our definition and sum up these ratings to obtain some 
< scaled value that v/ouTd reflect degree of individualization* We stated 

then, and believe now, that such a scale is continuous in nature and that 

t}ie obvious i/itent of the study requested by NIE to be designed ii> this 
' effort was to contrast programs from opposite ends of that spectrum in 

order to assess the effectiveness of individualization. The definitional 
\ matrix relied upon here isS'dentical in cojicept to this anticipated 
scaling a ppro ach .though different in appearaigce. If one assigns a value 

of +1 to each "yes" associated with one of the key issues reflected in that 
'matrix and 0 with each "no", the scaled total ratings for the four study 

groups would be as follows: 

Total Scaled Ratin g 

9 Group I "yes" to all four -issues 4 

d Group II "yes"" to any three issues ^,.3 

0 Group III "no" '.to any three issues 1 

0 * Group IV "no" to all four issues 0 

Thus, the four study groups may be viewed^s falling along a continuum of 
indi vidqalization and the omission of prograiDS with total scaled ratings 
of 2 is consistent \/ith our original concept of NIE's- desire to contrast 
programs from opposite ends of that continuum. The matrix display tech- 
nique was chosen because it illustrates the factorial nature of our approach 
• better than a rating sca^e would, and it iS felt that thts factorial 
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(or quasi-factorial) nature in the compression of the overall 15-cell 
matrix to the four study groups, allows much to 6e dpne in terns of " ^ . 
powerful analytical techniques.. 

A point made, earlier in the discussion of this definitional 
matrix in Task 2 warrants re-emphasis here-. The TURNKE^T study team has . 
concluded that based on the best informati.on at^ hand^and, knowledgeable 
estimates of anticipated information in this regard, the shaded area 
of the definitional matrix would be excluded from the analysis. This 
decision is obviously in accord with^ the discussion of the^ above para- 

•s. 

graph. However, in the interest of^horc^ughness it was pointed out ir\ 
T^sk 2 that it may be that filling in all cells of the definitional matrix 
is fe^asible a fact that might become apparent to the implementation 
contractor as further candidate sites are identified. If this is the case, 
in spite of any initial desire to observe programs only, from the ends of ^ 
the individualization continuum rather than from the middle ground, from 
a factorial design point of view it may be desirable to study auch programs 
in order to more fully investigate higher level interactions among the 
four key issues related in the definitional matrix. This deviation from 
the analysis plan suggested here is an. option that depends entirely on 
the availability of sufficient data for such middTe ground programs — 
the probability of which is assumed to be low by the TURNKEY study t?am 
at this time. 

IGD . . • • 
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Research Ques tions to be 'Addressed 

For ease of reference the basic design framework described earlier in 
detail in Task 2 is presented again here as Exhibit III-21. The individual 
cells are identified by the numbers 1 to. 15; the four study^groups of programs 
are identified as numbers I to IV. Exhibit III-22 repeats the specific four 
s'^mpling frames 5r repetitions of the basic framework iterated over the 
other sarrpliag dimensions , ('mains tream vs. pull-out and reading'vs. math) 
besides individual i^ation/standardizatioh. 

^This design framework allows a number of; research questions to be 
addressed. The discussion be-Tow presents these questions within a hierar- 
chial structure of 4 leyeTs. The Level 1 issues are those most basic to 
the -proposed study. Lefvels 2, 3, and 4 include issues that are j'mportant 

^ to address in the study, but the c/n tical ^alue of the issues addressed 
(and ability of the proposed design to provide pov^erful results regarding / 
these issues) decreases .at each step^ 'along this hierarchy. 

The questions presented below are to be answered by means of a series 
of con-trasts". The- specific data to be contrasted in order to' answer 
each specific issu'e could be any, if not all, of the dependent variables » 
presented in an earlier section^ excluding the '"other" variables discussed 

.at that time. Included in th'^ese possibilities is the additional procedure 
for meas-uring program-fair success discussed in detail in Task 4. In each 
case the results that are compared will be the combined results for that 
dependent variable from all second and third grade classrooms included in 
that specific sampling cell. Thus, the entire set of contrasts presented 



170 

III-130 



R\SIC DESICN rilAMEKQRK 





SPEC! PI C ' PERFOr0.lA\C4?'^OTJEOT 
.W: ASS I GXED TO/EACH STUDENT* . — 


SPECIFIC PERFORMANCE OBJECTIVES 
^<ARE NOT .ASSIGNED TO EAQl STUDENT 


DiAaxosis/ 
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IS CONTINUOUS. 


DlACts'OSIS/ 
PRESCRIFnON IS, 
NOT CONTINUOUS 


DIAGNOSIS/ 
PRESCRIPTION 
- 15 CONTINUOUS , 


DIAGNOSIS/ 
PRESCRIPTION IS 
NOT CONTINUOUS 
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INDIVIDUAI. 

PACING 

EXISTS' 
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GRADES: 

NOMINAL COMP ED INSTRUCTIONAL SETTING: 
SUBJECT AREA: 
UNIT OF ANALYSIS: 
PROGRAMS STUDIED: 



GRADES: ) 

NOMINAL COMP ED INSTRUCTIONAL SETTING; 
'SUBJECT AREA: 

mil OF ANALYSIS: » 
PROGRAI-'iS STUDIED: 



GRADES: 

NOMINAL COMP ED INSTRUCTIONAL SETTING: 
SUBJECT AREA:_ 
UNIT OF mUSlS: 
PROGRAMS STUDIED: 



GRADES: 

NOMINAL COMP ED INSTRUCTIONAL SETTP-'G: 

SUBJECT AREA: 

UNIT OF AriALYSIS: ' 

PROGRAMS STUDIED: 



2 and 3 Combined 
Mainstream 
Reading 
"Classroom" 

Groups I (n=51), II (51), 
III (51), and IV (51) 

2 and 3 Combined 
Pull-Out 
Reading 
"Classroom" 

Groups I (n=49), II (49), 
III (49), and IV (49) 

2 and 3 Combined ' 

Mainstream 

Math 

"Classroom" 

Groups -I (n=51) , II (51), 
III (51), and IV (51) 

2 and 3 Combined 

Math 

"Classroom" 

Groups I (n=49), II (49), 
III (49), and IV (49) 
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below 'bay be repeated for each of the dependent variables for which adequate 

i 

result'^ are obtained and which are considered relevant to the questions 
being ^tudied. Hov^ever, given the multi- face ted nature of the outcome measures 
and instruments described earlier in Task 4, a listing of 'these measures in 
the ordter of their anticipated imp^ortance for the 'contrasts outlined here 
is appropriate.^ This priority listing is as follows: 



1. 



2. 



Total scores , achievement "tests 
0 Reading ^' 
e Math 

'"Program- fair" measures of effectiveness, based 
on achievement scores ^ . 
0 Reading " ^ 

0 Math 

Total seal e,. scores , effective .instruments^ 
e Piers-Harris 

d atN 

0 OCI 

Sub-scale scores ^'achievement tests 
e Reading . ; 
0 Math 

Sub-scale scores* effective instruments 
G Piers-Harris' 
0 ATS 
e OCI 



ERIC 



173 

.111-133 



The specific means or statistical techniques used to assess each contrast 
presented below would range from simple comparison of means or frequencies 
via t-test or contingency table methods, where appropriate, to more sophisticated 
approaches such as ANOVA or discriminant function analysis. Where specific 
cells or groups of cells from the definStional matrix are contrasted, 
tec|^iques for unequal sample size treatment wil] be applied as needed. 

Level 1 Analysis is presented as Exhibit III-23. Level- 2 Analysis is 
presented as Exhibit III-24. Level 3 Analysis is presented as Exhibit III-25. 
Level 4 Analysis is presented as Exhibit III-25, The approximate number of 
classrooms involved in each contrast are also sbown. Note that where these 
contrasts involve individual cells in the IS-cell Definitional Matrix 
other than cells 1 and 16, the number of classrooms indicated is strictly 
an approxirpat/on based on an assumption of a uniforiii distribution of classrooms 
within stjpdy groups II and III in this matrix. This assumption may be 
faulty giVen the preceding discussion on this issue, but as a means for 
inustrating\he scope of the contrasts, it is felt that the assumption is 
acceptable ir/ this case, 

. Addit jonal Analysis 



Beyond the detailed lists of contrasts described in Exhibits III-23 
through III-26, a number of potentially critical analyses will.be addressed 
under the heading of Level 4 Analysis. By far the most critical and 
for now identifable of these additional analyses is the opportunity to 
cross validate the results of TURNKEY'S efforts in comp ed cost effective- 
ness analyses conducted elsewhere, specifically in Michigan. This possibility 
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EXHIBIT III-23 



LEVEL 1 ANALYSIS 



What is the impact of individualizatign on- comp ed program effectiveness? 
Relevant Contrasts: 



(All programs 
studied) 



(Reading Programs- 
only) 



1. ' Group I (A, B, C, and D combined, n=200) vs'. 

Group IV (A, B, C/and D combined, n = 200) 

2. (Group I (A, B, C, and D combined) plus 

Group II '(A, B, C, and D combijied, n=400) vs. 
(Group III (A,-^, C, and D combined) plus 
Group IV (A, B, C, and p combined, n=400) 

3. Group II ^(A, B, C, and D combined, n=200) vs. 
Group III (A, B,'C, and D combined, n=200) 

4. Group I (A and B combined, n=ldO> vs. Group IV 
(A and B combined, n=100) 

5. (Group I (A and B combined) plus>Group'II 

(A and B combined), n=200) .vs. (Group III (A 
and B* combined) plus Group IV -(A and B ' 
combined), n=200). . - ' 

6* ^ Group II^(A and B, combined, n=100) vs. ^Group III 
(A and B combined', n=100) 



(Math Programs 
only) 



7. 
8. 



9. 



I Otip 1 \\j uMu u cuiuDiiicu, n-iuu; v:>. 

(C and p combined, n=100) 
(Group r (C and D combined) plus Group 41 
(C and D Cof^bined, n=200)'vs. (Group III (C and . 
D combined) plus Group IV (C and D combined, n=200) 
Group n (C and D combined," n=^00) vs. Group II r 
(C and D combined, n=100) 



1 ^^^^ 
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EXHIBIT irN24 
Page 1 of 3 



LEVEL 2 ANALYSIS 



I. What is the impact of each of the specific processes included 
in the study's definition of individualization on comp ed • 
program effectiveness? 



Performance Objectives? 

Relevant Contrasts (recalling the 16 cells iri the definitional 
matrix and. the specific cells included in eacli of the four 
study groups of programs):. 



(All cirograms 
sty^fied) 



(Reading 
Programs 
only) • 

(Math 
Programs 
only) 



1. Cell 1 (A, B, C, and D combined, n=200) vs. 
Cell 3 (A, B, C, and D combined, n=50) 

2. Cell 16 (A, B, C, AND D combined, n=200) vs. 
Cell 14 (A, B, C, and D combined,- n=50) 

3. Cell 1 (A and B combined, n=100) vs. Cell 3 
(A and B combined, n=25) 

4. Cell 16 (A and B combined, n=100) vs. Cell 14 
' (A and B combined, n=25) ^ 

5. Cell 1 (C and D combined, n=l'00) vs. Cell 3 
(C and D combined, n=25) 

6. Cell 16 (C and D combined, n=200) vs. Cell 14 
(C and D conibined, n=?5) 



Continuous Diagnosis/Prescription? 
Relevant Contrasts: 



(Al 1 programs 
studied) 



(Reading 
* Programs 
only) 



(Math 
Programs 
only) 



1/ Cell 1 (A, B, C, and D combined, n=200^ vs. 

Cell 2 (A, B, C, and D combined, n=50) 

2- Cell 16 (A, B, C, and D combined, n=200) bs. 

Cell 15 (A, B, C, and D combined, n=50) 

3. Cell 1 (A and B combined, n=100) vs. Cell 2 
(A and B combined, n=25) 

4. GelM6 (A and B combined, n=100) vs. -Cell 15 
(A and B combined, n=25) 

5. Cell 1 (C and D combined, n=100) vs. Cell 2 
(C and D combined) . 

6. Cell 16 (C and D combined, n=100) vs. Cell 15 
(C and D combined, n=25) 
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EXHIBIT III-24 
Page 2 of 3 



Alternate Learner Paths? 



Relevant Contrasts: 



(All programs 
studied) 



(Reading 
Programs 
only) 

(Math 
Programs 
.only) 



1. Cell 1 (A, B, C, and D combined, n=200) vs. 
Cell 9 (A, B/C, and D combiaed, n=50) 

2. Cell 16 (A, B, C, and D combined, n=200) vs. 
Cell 8 (A, B, C, and D combined, n=50) 

3. Cell 1 (A and B combined, n=2O0) vs. 
Cell 9 (A and B combined, n=25) 

4. Cell 16 (A and B combined, n=200) vs. ^ 
Cell" 8 (A and B combined, n = 25) 

5. Cell 1 (C and D combined, n=100) vs. 
Cell 9 (C and D combined, n=25) 

6. Cell 16 (C and D combined, n = 100) vs. 
Cell 8 (C and D cofnbined, n=25) 



Pacing? 

Relevant Contrasts: 



(All programs 
studied; 



(Reading 
Programs 
only) 



1. Cell (A, B, C, and D combined, n=200) vs. 
Cell 5 (A, B, C, aijd D combined, n=50^) 

2. Cell 16 (A, B, C, and D combined, n=26o) vs 
Cell 12 (A, B, C, and D combined, n=50) 

3, Cell 1 (A and B combined, n=100) vs.* 
Cell 5 (A and B combined, n=25) 

4, Cell 16 (A and B combined, n=100) vs. 

Cell 12 (A and B combined, n=25) , 



(Math 
Programs 
only) 



5, Cell 1 (C and D combined, n=100l vs. 
Cell 5 (C and D combined, n=25) 

6. pell 16 (C and D combined, n=100)'vs, 
Cell 12 (C and D combined, n=25) 



II. What is the impact of having all but ojie of these processes 
present (no matter v/hich one*is lacking) on comp ed program 
effectiveness? 




Relevant Contrasts: 



(Al 1 Programs 
studied) 



Group I (A, B, C, and D combined, n=200) vs. 
Group II (A5 B, C, and D combined^ n=200) 
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(Reading Pro 
grams only) 

(Math Pro- 
grams ^nly) 



- 2. Group I (A and B combined, n=100) vs< 

Group II (A and B combined, n=lX)0) 

ST Group I (C and D combined, n=100). ys< 

Group II (C and D combined, n-100) 



What is the impact of lacking all but one of these prpcesses 
(no matter which three are lacking) on comp ed program 
effectiveness? ^ 

Relevant Contrasts: 

(All Programs 1. Group III' (A, B, C, and D combined, n^O) 
studied) Group IV (A, B, C, and D, 'combined, n=200) 

* ^ ■ 

(Reading Pro- 2. Group III (A and B combined, n=100) vs. 
grams only) Group IV (A snd B combined, n==100) 

(Math Pro- 3. Group III (C and D combined, n=10O) vs. 
grams;only) Group TV (C and D combined, n=100) 
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EXHIBfT III-25 



jWhat is the impact 
on comp ed prograni 



Level 3 analysis 



of classroom setting (mainstrjsam vs, pull-out) 
effectiveness? 



Relevant Contrasts; 



(Reading and 
math combined) 


• .1.' 


(^up 
Group 


"(Reading.,, 
only) 


2. 


Group 
Group 


(Math 
only) 

> 


V 

' 3. 


Group 
Group 


(Reading and 
math combined)^ 


4. 


Group 
Group 


(Reading 
only) 


5. 


Group 
Group 


(Math 
only) 


6. 


Group 
Group 


(Reading and 
math combined) 


7. 


Group 
Group 


(Reading 
' only) 


8, 


Group 
Group 


(Math 
-^only) ■ 


9, 


Group 
Group 


{Reading'and 
math combined) 


TO. 


Group 
Group 


(Reading / 
only) ' 


11, 


Group 
Group 


(Math 
only) 


12. 


Group 
Group 



(A and C combined, n=102) vs.. . 
,(B and D combined, n=98) 

(A, n=51 ) vs. 
(B, n=49) 

(C,'n=51) vs." 
(D, n=49) 

(A and C combined, n=102) vs. 
(B and" D- combined, n=98) ' 

(A, n=51) vs. 
(B, n=49) 

(C, n="51) vs. 

(D, n=49) ^ • \ 

I (A and C cor'bined, n=102) 
I (B and D combined, n=98) • 

I '(A, n=51) .vs. 
I (B, n=49) 

I (C, n=51) vs'. 
I (d, nM9) 

V '(A and C combined, n=102) vs. 

V (B and D combined, n=98) 



V (A, n=51)'vs. 

V (B,. n=49) 

V (C, n=51) vs. 

V (0, n=49) 



< 
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EXHIBIT III-26 

1 



LEVEL 4 ANALYSIS 



What is the impact of other program factors beyond the processes 
and settings' treated in Level 2 or Level 3 analyses on comp ed 
program effecti-veness?- 



Level 4 analyses are to be of a general exploratory nature 
using data gathered on either the covariables or the "other" 
variables discussed in the initial section. Potential relation- 
ships between observed data for these variabl es *and program effective- 
ness will be investigated using the appropriate available techniques- 
Specific subsamples of very successful Group I or II programs and 
very unsuccessful such programs could be^ selected and contrasts 
then be drawa on the values of the variables^being expl.ored 'between 
these two groups. Similar samples could be drawn from the .progran^s 
in Groups III and IV. In this v/ay^ the specific classroom or progra;r. 
management characteristics or student factors that seem tb'be. 
associated w:^^h success in each of ±he types of programs could be 
identified. It is recognized that such an exploratory analysis would' 
at best only specify a list of factors to be consolidated i,n ^sub- 
s-equent research, but if this list is generated by the' proposed study, 
the effect would be well worth even this interim accofnpl ishment. 
Th,is particular option wiUte explaiited in more detail tn this section 



iir-r4 



was discusseji briefly in Task 1 where it was mentioned that, although such 
an approach would be basically exploratory in nature and results would demand 
cross validation elsewhere if it were approached in a strictly exploratory 
manner, by viewing the cost-effectiveness results to date, derived from / . 
application of the same TURNKEY instruments suggested Jor use here, thts 
particular analysis could actually' serve as a cross-validation effort 
of these existing results. TURNKEY'S work on the Michigan Cost-Effective- 

^ ness, Model will have a series of results available for consideration in 
such a cross-validation effort well before the Implementation^of the study 
proposed here, thus facilitating the finalfzation of this specif tc^ substudy 
once the impleR3entat1on contractor begins work on this study. 

The effectiveness portion of this subanalysis would likely take the 
form of examining the relationship between program success and, the various 
^ program characteristic's that a-volve from the current TURNKEY work in 

Michigan to determine whether these same relationships obtained on the sample 
of programs examined in this study. Perhaps even the path models developed 
\x\ the current TURNKEY work could be examined for cross-validation purposes 
within the NIE sample, taking this effectiveness component beyond the level 
of more simple contrasts. - . , 

^ The cost portion this subanalysis v/ould involve an application of 
TURNKEY'S widely used cost analysis methods, based o|^ the COST-Ed"*"^ Model. 
Exhibit III-27 displays the basic calcplation stream involved in this 
analytical method; Exhibit 111-28 is. an example of th^ specific program ' • 
cost results obtained from a comp ed program similar, to those to^be studied 
here; and Exhibit III-29 is a further example of the cost analysis results 
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Exhibit III-27 




ac 06 ac 
S * O -o 
to &. < 



S 
i 



/ 



o o 









< su 5 




* J .1 

§ o > 


i 



H ^ 

as r> 

w 

2 as 

5 s: 0 




X ^ 2 ^ 
^ •> C 

=3 3: •< 



% 



31 C 3 u O 



Si r. 






3 




> 
:^ J 












>- 


Lj ^ o 
o > — < 



ERIC 




• 
• 
• 



182 




III-142 



EXHIBIT III-28 



DOLLARS PER COMP-ED STUDENT ANNUALLY " 



SITE 1116 



RESOURCES 



FUNCTIONS 



\ ►-I 

2: < 
O LU 



LU »—« 

O -J 



LU I— « 
I Z 
CL I— < 

o ci: 



I 

cc: 

Q O Q 00 
UJ 1— ' O LU 1—1 

I 00 2:1 I z 

Q_ I I— i.a. 1— I O 
O LU ct;0 Q 1— 



Resource 
Total 



Percent 
of 
Total 
Cost 




PERSONNEL 
District Conp-Ed Director 
Principal 
Comp-Ed Teacher 
Regular Teacher 
Paraprofessional. 
Reading Specialist 
Other Cl'assroom Staff 



CONSUMABLES' 

Comp-Ed Books and 

AV Software 
Regular Books and 

AV Software 



EQUIPMENT 
Comp-Ed Av Equipment 
Other Comp-Ed Instructional* 

^Equipment 
Regular AV Equipnent 

!r Instructional Equipment 
Comp-Ed Administration 
Equipment 



92 
63 
82 



11 

23 
21 
151 



27 



MISCELLANEOUS 

Miscellaneous Comp-Ed 

Training Expenses 
Miscellaneous Conp-Ed 
Administrative Expenses 



5 

.4 



12 
34 
4 
7 



12 

2 



35 
64 

121* 
221 

• 82** 



6.3 
11 .4 
21 .6 
39.5 
14.7 



27** 
6 . 



4.8 
1.1 



0.2 



2** 



0.4 



FUNCTION TOTAL 
PERCENT OF TOTAL COST 



271 
48.5 



206- 
36.9 



1.6 



9'> 57 
10.2 



16 
2.9 



•559 ' 
100.1 



100.0 



^ All or part of these totals provided from Comp-Ed Budget 
**A11 of these totals provided- fronj^&omp-Ed Budget 
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available from the application of these techni^ques — in this case the COST- 
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ED report known as the Economic Factor Ranking. This Ranking listl^jhe c^st 
factors which are part of the cost structure of any given program in the ♦ 
* order of their impact on overall program cost, with those i tents K^an^ed at 
the top of the page having the greatest impact and those near the bottom, 
the least. Such reports as shown in .Exhibit III-28 and Exhibit JII-29 
would be available for each program/classroom studied allowing these 
cost results to be contrasted along with the outcome results to be 
analyzed as described above. Further, such cost results could even be 
considered together with the program-fair scheme for program comparisons 
described in Task 4. 

i 

A major consideration to be addressed in the cost portion of this 
subanalysis is the issue of standarjJ pr;icing or local pricing. In question 
here is whether to adjust priqes for varicJus educational resources (e.g., 
teacher salary) to a common standard for this nation or for some other 
regional unit larger than the LEA. A teacher with a M.A. degree and six 
years of experience, for instance, wilTreceive a vastly different salary 
if he worked in a large urban center in the Northeast than if he worked in 
a small rural district in Texas. Differences in the cost of living, the 
impact of local labor supply/demand conditions, and the power of local' teachers 
organisations are just a few of many reasons for. such differences.- In a 
study such as that proposed here 'that is national in scope, one could argue 
for the use of standard pricing so that differences in cost;s accurately 
reflect differences in the amounts of resources consumed rather than 
simply differences in prices. , ' . ^ ' / 
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On the other- hand, local pric6/salary structures undoubtedly influence 
the actual mix ofresources found in any given program. The trade-offs 
among educational resources at two different sites using ostensibly 
the same program will vary depending upon the relative price of these 
resources; where aides -are relatively less costly with respect to 
certified teachers^ more' aides will be used and where the opposite is true' 
regarding reilative prices the opposite mefy well be true regarding the re- 
liance on a^es. Given this interaction of the local prjce structure with 
the'^ctual configuration of the program as operated locally and .the resource 
mix nature of individualized instruction in general, a stronger argument 
could be made for the use of local or actual prices in the cost analysis. 

COST-ED can be used in either a local price mode or a standard price 
mode so that, whichever argument is found to be most pursuasive,COST-^ED analysis 
can readily accommodate either option. A third option that of computing 
both local and standard price-based costs is also readily accommodated by 
COST-ED. Other resource/cost models currently in use (e.g., the Haggert 
Model, the RMC Model) can be exercised more .easily in the standard price 
mode ^han^ in the local price mode. 

Beyond its flexibility regarding the issue of standard pricing. 



COST-ED' 'lends itself to the cost portion of this subanalysis for a number 
of other reasons : 

© It is flexible and adaptable to any type of instructional , 
program (e.g^, reading, math), classroom setting (e.g., 
main's tream, ^)ull-out), and program organizational level 

(e.g., classroom, school, distr,ict-wide) 
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It directly reflects the impact of the length of the 
program (i.e., number of hours of instruction per student) 
on the cost of the program- . 
It provides, opportunities for conducting trade-off 
analyses of various configurations of a specific 
program. ' - ' , 

It is useful to analysts in asspciating costs to 
program features found to be related to progfant 
success. ' • , ' \ 

It is a potentfalTy useful incentive for LEA cooperation, 
since the model can provide useful^data ^o local managers 
through s.ensitivity analyses. Such analyses cap .help in 
idejTtiJ^Ing alter^native ways to cut costs' if the manager, 
'is faced with budget cutbacks or in allocating additional funds 

^ . ■ r 

if the opposite is true., j 



Treatment of ItnpTementation Results \ 
It is suggested that implementation data be used in a number of wa^ys. 
Tir^t of.ctll, tho^e classrooms not considered v/ell implemented (thcjugh ^ 
fewkre expected to fa1V*through the sarffple selection process described 
earlier) -v/ill be dropped from the contrasts that are included in Levels 1 
2y and 3. ^ T-his aqtion will undoubtedly result .in unequal, n contrasts in 
most if not all cases; appropri-ate techijiques toVjeffect these unequal 
n situations will be applied, as needed. ' 
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Second, the impact qf implementation considerations on program effec- 
tiveness could be as^esse*by considering the differences in effectiveness 

4 

and degree of implementation among classrooms within any, given program, / 
This analysis would be facilitated by the existence of a uniform instrument 
used for all classrooms part of the same program. Among programs', however, 
this relationship would be more difficult to assess to the extent that the 
implementatiory checklists and other assessment instruments vary from program 
to program. .To the extent that such among-program degrees of .implementa- 
tion assessments can be 'made, this factor will also be analyzed. All 
classrooms studied will be part of the analysis cqnducted for this particular 
issue, " ' 

Data f^eduction ^ 

Data obtained.'throughout the 1976-77 school year* from pre- test admir^is- 
tration and from periodic observatian of program operation and character- 
istics will be reduced for analysis upon collection. Specifically, data ^ 
obtained from pre-test, initial interviews, a.nd the initial observation cycles 
at all sites will be processed in a manner that will allow the data reduc- 
tian and analys^is procedures to be evaluated early during the study year. 
Procedures for the speedy reduction of the post-test results and other late 
school year data will be carefully refined during this school year to ensure 
that once the data are obtained! in' May 1977 all necessary data are reduced 
and ready for analysis in order to report the study^s results'in July 1977, 
Since TURNKEY e'xperience has indicated the extremely cr^itical nature of these 
'data -reciuction tasks in' the past, sufficient resources will be allocated 
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to this activity in our proposed 'design to ensure thetr timely completion. 
Once all data has been obtained, « the carefully planned data analysis 
effort will proceed. The practical details of this critical element of this 
study are addressed further in Section IV of this report. Appendix 13 
includes examples of the data coll ectiori/coding/reduction forms modified from 
forms used by TURNKEY elsewhere. 

Implications' and Limitations of the Approach 

The nature of the sampling plan, especially its r:eflection of our. 
Definitional Matrix, allow a number of issues critical to NIE and the prime 
audience of the proposed study (e.g., Congress) to be addressed in a highly- 
structured manner. Though a true experiment wherein all else was kept 
constant and on,ly one program feature was systematically varied at any 
given time and students were assiyfieu Lu treatmerits randomly is impossible 
to achieve in any real world setting, the qua'sirfactorial structure 
suggested hereallowsan approximation of at least 'a portion of this scheme 
^to obtain. (The reason for not fully utilizing the factorial -irgf&ra'ction 
" potential. of our Def ini.tional Matrix was discussed .in detail earlier^.) 
Careful selection of progi^ams and matching of students in adherence to' the 
sampling constraints 1 isted earlier will have to suffice' as the suggested 
approximation of random assignment of students, recognizing that this 
approximation 'has been a major problem for all large-scale field research, 
and evaluation studies in education. - * ^ * 

The analyses and contrasts^ delineated abovfe should provide adequate 
ansv/ers 'for NiE to meet its obligation to Congress and for Congress to 




effectively tackle the task of writing the new comp ed legislation in. 1977, 
bther contrasts arje clearjy available as well given the proposed design 
framework. For instance, the issue of whether the processes included in " 
our definition of individualization have different effects in mainstream 
vs. pull-out programs could clearly be investigate(^ by combining the'togical 
framework of Level 3 with that of Level 2, Only the contrasts considered 
iTiost basic have been specifically listfed here, all other* possibilities 
remain for the research team 4:0 consider throughout the implementation 
phase. ' . 
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SECTION IV ' 
LOGISTICAL ISSUES/TASK APPROACHES ' 

In Section III, we have specified the theoretical afi^^'ana^ 
framework within which the implementation of the 'individualTzed instruc 
tion study will be conduct^. In this section, we describe the opera- 
tional aspects of the study and its interaction with the participating 
programs. Specifically, we address the d^ta collection procedures to 
be used (Task 5), techniques for enlisting site cooperation .(Task 7), 
implementation project staff (Task .8), and the overall project schedule 
(Task 9). * • - 
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TASK 6 - DEVELOP DATA COLLECTION PLAN 

In Tasks 1 thraygh 4, discussed in Section II of this report, we 
have performed the first preliminary data collection step the Identi- 
fication of the study's data (and hence ,data collection) requirements. ^ 
As in'dicated in our Proposal, these needs fail] into three principal 
categories : ^ ' 

• ' Program and School. Characteristics management and opera- 

tiOj4i$l factors. 

• Program Irnplementatipn Information level -of-implementation 
data on the degree to which the study programs are operating 
as intended , , , 

Program Effectiveness Measures program outcome measures 
of student performance and attitudes 

Also tn the first four tasks, we discussed the detailed factors involved 
i,n the development of the specific instruments to be used to gather the 
needed ,udLd. In Section Ill/Jask 5, we specified the procedures zo oe 
used to'; analyze the' data, dnce it has been collected. In this task, we 
outlined the data collection plan the logistical procedures which 
bridge the gap between the data .collection instruments and the analysis 
plan. ' ' , • - 

The data collection plan for this study of individual ize'd instruc- 
tion has five major tompoaents: > 

' 1) * Selection of Field Staff ' ' . 

2) /Training of Field Staff 
*3) Data Collection Procedures 

4) Field Cooperation ► 

5) ' Data Confidentiality' 



The first two of these components relate specifically to the staffing 
of the project and will consequently be discussed in Task 8, the de- 
velopment of a staffing plan. The fourth component, gaining the coopera- 
tion of school district personnel at each site, will be discussed as an 
individual topic in-Task 7. In this task we will, therefore, discuss 
only the data collection techniques to be used and the procedures to be 
followed in ensuring data conf identali ty. , ^ 

Data Collection Procedures 

As indicated in Section Ill/Tasks 1-4, data collection activities ' 

s 

will consist of three general instruments/techniques: 

• staff interviev/s 

• testing and affective measurement 

• classroom observation 

The ddtd collect TOP p^^oce'ciures v/ill reflect the nature of the dutu collec- 
tion instruments/techniques to be used. After preliminary subtasks, in 
which the instruments are finalized, other preparation activities con- 
ducted, and the instruments/techniques field tested, interviews of school' 
di.^trict staff pmbers will be conducted, pre- and post-tests of students 
will be^ administered, and classroom observations will^ be performed. Each 
of these specific subtasks will be described in turn. It should be noted 
that it is possible to specify the procedures to be used although some 
of the/p^ruments themselves may not be finalized. Some of the proce- 
dures described herein will be of a general nature, to be finalized when 
the first data collection subtask, the finalization of all data collection 
instruments is conducted as indicated jn Section Ill/Task 3". 
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Final izatlpn of Data Collection Instrume nts 
■■ — J — 

Since timing is' a critical factor in the performance of this study, 
it is important that the instruments to be used in the collection of data 
be finalized as early as posstbTe so that the necessary and time-consunjing ' 
0MB clearance procedures maybe completed without delaying the project. 
For all .intents and pui^poses,' it is possible to divide the recommended 
instruments (see Section Il'l/Tasks 1-4) into two categories: those which 
may be final ked "^before site selection with very little work on the part 
of the implementation contractor, and those which are site-dependent and 
'wHich must consequently await final site selection. 

;Most of-the instruments fall into the first of these categories. 
Achievement tests and specific affective measures (discussed in Section 
Ill/Task 4) ar'e all to be used .in their existing forms and, therefore, 
will requi.i^e no modification (some modification in traditional scorlrig 
^nd analyses procedures may be needed, but these would not affect the 
form or administration of. these existing instruments). The interview 
guides for interviews of principals, teachers, aides, and other staff mem- 
'\?^rs have beea specified, to some degree of detail in Section Ill/Tasks 1-4 
(an example of 'a finalized interview instrument iS' shown in Appendix 13) 
and will .require only minimum modification by the implementation contractor 
The m'odificatibh of these instruments should take no more than eight weeks 
lifter implementation' contract award and will, therefore, be ready for sub- 
mission to 0MB for approval and field testing in selected school districts 
d^ur'ing Apt'il 1976.' ^ 
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the suggested' data collection instruments and submit them to 0MB prior 
to the selection of the implementation contractor. We believe, however, 
that it is desirable for the implementation contractor to have some input 
into the instruments to be used. This delay until Spring 1976 for 0MB' 
submission will not delay the project in any way since it does not fall 
on the 'study's\critical path. All instruments will be ready far use when 
school beg'ins in September 1976. 

Field Test of Data Collection Instruments 
■ , During the Spring of 1976, the data col lection*instruments and pro- 

. cedures, will be field tested in two school districts. 

These field tests will be limited to those instruments which have 
been modified or developed for this study. The student outcome instru- 
ments, the achieveiTicrit and dffe^-tive medsures need not be inciuaed in the 
field test for two reasons: 1) the administration of achievement and 
affective tests to students at field test sites might unduly burden the 
classroom staff, and; 2) each of the suggested student ojjtcome instru- 
ments has already been field tested and validated many times before. 

Specifically to be included in the field test are the interview - 
instruments for principals," teachers, aides, and other staff members, and- 
the classroom observation instruments. Th« field test of the classroom 
observation instruments implicitly denotes a field test of the procedures 
by which the classroom observation' instrument/implementation checklist 
is determined for each program (see Section Ill/Task 3), 
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It is suggested that one individualized and one standardized pro- 
gram be selected in each of the two field test districts. One of the 
districts should have two reading programs, while the other should have 
two mathematics programs. Within each of these programs, the recommended 
.field test will include interviews with the school principal, at least 
"two teachers (including at least one regular classroom teacher who serves 
compensatory education students plus at least one specially hired com- 
pensatory education teacher), and at least one aide (if applicable), as 
well asy^of the district-level Federal program director. Also within each 
of the two programs at each field test site, at least three classroom ob- 
servations will bexonducted. 

Sites for the field tests should be drawn from the list of programs 
suggested for inclusion in the study. This will ensure that the field 
tes-'t sites not only have an individualized instruction program, but that 
enough information will be available for the development of classroom 
observation instruments/implementation checklists for each program. For 
purposes of economy and usefulness, the two field test sites should be 
relatively close to the implementation contractor's major office and should 
differ 'somewhat in the general nature of their population. 

The logistics of the field test indicate that plans for its conduct 
should be undertaken inufiediately after implementation contract award. A 
member of the inplementation contractor's project team should be designated 
as Field Test Director. ^ This Field Test Director will be responsible for 
contacting each of the field test sites and arranging for their cooperation 

6 
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in the field test. This field test director will s'erye cts continuing liaison 
between theTield test sites and, the impleinentation contractor, and may even 
administer the field tests of specific instruments personally. 

It is expected that three principal benefits will emerge from the 
field test. The first is the obvious benefit of being able to provode 
operational support for the 0MB documentation package. The second, less 
obvious, benefit is the insight gained into the procedures associated with 
the administration of many of the data collection instruments. It is. pos- 
sible to capitalize on this insight by making the Field Test Director an' 
integral part of th^ project group which plans and conducts the training 
sessions for data c6llectors. A third, and even less obvious, benefit pf 
the field test should be an understanding of the incentives which might be 
needed to ensure the willing participation and cooperation of district, 
school, and classroom staff. 

Preparation. for Data Collection 

An often overlooked but extremely important aspect of any major data 
collection effort is the laying of groundwork for-^ proper communication and 
coordination throughout the project. The purpose of this subtask is to 
keep appropriate groups and individuals abreast of the project and to en- 
sure that scheduling of project activities is performed as expeditiously 
as possible. ^ ' 

It is recommended that before specific contact is made with prpject 
sites, the implementation contractor contact the Committee on Evaluation 
and Informatioft Systems^ (CLIS) of the Council of Chief State School Officers 
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(CCSSO). Not only will this initial contact satisfy protocol, but it 
is quite possible that SEA staff may be able to suggest addi.tional well- 
implemented individual ized. programs to be considered as project sites. 

The first contact with each of the districts after their selection as 
project sites should be by telephone using the LEA official designated in 
the sampling questionnaire as the appropriate contact. In most areas, * 
this contact person will be the district's Director of Federal Programs. 
This initial s'ite contact should take place immediately upon the selec- 
tion of each site for inclusion in the final sample, during April and May 
1976. The purpose of this initial telephone contact is: 

• • to inform that district of its selection and confirm its 

participation desire; 

• to formalize the communication channel between the district 
designee and the Data Collection Manager of the implementa- 
tion contractor; 

f to specify the schools and classrooms which have been chosen 
as study units; ? 

• to offer'' to assist in explaining the project toMocal groups 
(e.g., parents), through on-site presentation; ' 

e to negotiate the specific incentive arrangements to be used 
to facilitate staff cooperation (see Task 7); and 

0 to solicit district input into the selection of on-site data 
collection staff. 

It is believed that by enlisting -the district's designee as a partner in 
the data collection effort, the implementation contractor will minimize 
the probability of major disruptions during the course* of the project. 

After contacting the district-level designee, the implementation 
contractor should make initial contacts with the principals of the schools" 
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in which data is to be collected, lliis contact might best be made in 
two Steps: first, an initial telephone contact by the implementation 
contractor's Data^Collection Manager (see Task 8), informing the principal* 
of' his school's selection and introducing the local data collector; and,* 
second, a brief visit to the school by the local data collector, enabling 
him/her to become familiar >/ith the school, to establish open comrauiilca- 
tion with the principal, and to tentatively schedule time for the conduct 
of initial staff interviews and the administration of pretests for achieve- 
ment and affective instruments. 

It IS expected that a second preliminary school visit will be made 
by the local data collector during' the district's teacher orientation week 
(generally the last week in August), This visit will enable the Jocal data 
collector to introduce him/herself to the .teachers whose classes are in- 
volved in the study, to explain the overall nature of the study, to re- 
spond to any concerns the^^chers may 'have about classroom disruptions, 
and to confirm the schedule for interviews and test administration. This 
visit may involve meeting with teachers either individually or as a group, 
depending on the preference of the local data collector, the principal, 
and the teachers themselves. 

Although M has not been listed here as a specific preparatory ac- 
tivity, budgetary provision should be made for at least one additional 
(beyond the sampling visits) visit by a member of the implementation con- 
tractor's core project team. Experience tells us that such a visit can 
do much to facilitate the liaison between the local data collector and 
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site personnel. Althoug^such a visit may not be'necessary in all cases, 
it will frequently serve as healthy reinfarjncement for the initial train- 
ing session attended by. the local data^ collectors* 

As a final preparatory step, arrangements for computer facilities 

■J ^ • .■ ■ 

will be Riade. The importance of the time factor makes it imperative that 
rapid data turnaround be available. For tViis reason^the DHEW computer will 
not be used. A primary and backup facili.ty will be established to provide 
insurance againsf last^minute logistical* problems.' 

Conduct of ttaff Interviews 

The ftrst data col lection 'activity to take place- is the interviewing 
of school and classroom staff. These interviews should, if possible, be 
conducted during the first four weeks of the 1976-77^ school year. Indeed, 
it may be possible to conduct some or all of these^'nterviews in corijunc- 
tion with the introductory school visits just described. It is important 
that these interviews be used not only to gather data, but also to estab- 
lish a cordial relationship with teachers, which will facilitate the later 
administration of tests and conduct of Classroom observations. 

In order to reflect any changes in attitudes or perceptions on the 
parts of school personnel, the same or parallel interviews will also be 
conducted at the end of the 1976-77 school year. 

Although the specific procedures to be followed in interviewing 
school personnel will be different at each site, some basic precepts, 
gained from years of such interview experience, may be stated. When- 
ever possible, times for interviews should be scheduled at least two days 
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(and preferably a week in advance). Although principals generally have ■ 
the flexibility to be available whert needed for interviews, they are fre- 
quently interrupted by day-to-day school problems; it is generally advis- 
able to allow extra time (at least 60 minuites) for such principal inter- 
views. Teachers are generally available to be interviewed during thei-r 
free period if it is schedule.d in advance. The teacher's lounge or an 
empty classroom i? frequently the most suitable place for" such interviews. 
S\t)ce aides ;have less free (or planning) time than teachers, it is gen- 
erally more difficult to interview them. If properly scheduled, however, 
it is possible for the teacher to arrange the aide's time in such a way 
as to permit an interview during the regular school day. 

. . In terms of the time required of the local data collector for staff 
'interviews, the following estimates have been made: 

• Approximately 40 classrooms in each district will be in&luded 
in the st^udy sample. 

• Approximately 10 principals, 60 teachers, and 20 aides will be 
interviewed .in each LEA. r 

• About five interviews can be conducted each day (allowing 
for difficulties in scheduling interviewees): 

• The series of interviews will be conducted once at the be- 
ginning and once at the end of 1976-77 school year. 

The net result of these estimates indicates that between 35 and 40 days 

will be spent'by the local data collector i'nterviewi ng school staff. Ex- 

perience also indicates that an additional 15 days will be needed to code 

the resulting data and convert into a form suitable' for input into the 

analysis plan. In order to reinforce the training sessions, it is planned 
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that the .cognizant Regional Coordinator will provide on-site assistance , 
to the local data collector in' the conduct of early interviews,' This 
type of monitoring will also serve to enhance the cross-site reliability 
of interviews. 

Administration of Tests - - ^ * 

The second data collection activity is the administration of the 
achievement and affective test instruments specified in Section Ill/Task 4. 
In order that the results of the study maintain credibil i ty/o*t is critical' 
that the test results avoid contamination by inadequate testing procedures. 
Since all testing instruments are already developed, test administration 
guides also exist for all of them. These test administration guides will 
b,e discussed between the local data collector and each teacher (This dis~ 
cu<;sinn may takn place during the interviow.O, cir^d .the appropriate schedule 
for test administration will be developed. In' order to avoid variations in 
the time between pre- and post-tests, it is desirable that corresponding 
tests be administered at approximately the same time -.- all within a one- 
to- three-day period — following the suggested time periods in the manuals 
which are found on the "norming" test schedule. 

It is estimated that approximately ten working days will be required 
of the local data collector for the scheduling and administration of the 
pre-test, wilji a like time require for the post-test. In order. to conduct 
the testing in the manner described, the Regional Coordinators will a5|sist 
the local data collectors and the teachers during the pre- and post-te^t 
phases of data collection. 

4 
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Conduct of Classroom'Obseljvation . ' 

Frequent observations of all study classroom will be condueted during 
the months of September 1976 through April 1977. (These classroom obser- 
vations wi11.be C9nducted on an irregular basis but sqlieduled approximately 
two days prior to the visit.) - * 

The principal purpose of the classroom observations is to assess the 
degree to which the programs are .implemented accordit^g t6 their plan and 
to verify certain time schedules provided by teachers during these .inter-; 
viev/s. The specific 1nst>:uments to-be used in recording the class.room ob- 
servations will, in part, be uniquely determined for*€ach program, based . 
upon the specific goals, objectives, and procedures identified for the 
program. These instruments have "been discussed in Section Ill/Task 3. 

In terms of local data collector time required for classroom. obser- 
vations, we have made the following approxirtiations : 

0 All observations will be for an entire period. 

• There will be about 40 classrooms in each district 
participating in the study. 

• All classrooms will be observed four times from 
' September 1976 to April 1977. 

• No observation of a classroom will occur within 
30 days of the previous observation of that 

classroom. : 

• An attempt will be made to conduct the first class- 
room observation as early as possible after the pro- 
gram stabilizes generally around the third week 
in September. 

§ It is possible to conduct approximately four; full-- 
period classroom observations per day. 

« * 
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These approximations . indicate that about 40 days, during* the; pi^oject 'will 
be'spent by each local data collector 'in'' cjassrdofn. observation;^ Experi- 
ence also, indicates that an additi:6hal ten days w^ill -iSe needfed.for coding 

and other administrative tasks directly .ass^dciated- with ^he classrpom ob- 

i / \ " ' . ' ' , / * /. ^ 

servationS'. Again to ensure observation reliab)nit5 and to reinforce the . 

" ^ , w* • > ' * ^ ^ ' • 

training sessions, the Regional .Cpordinators wiTl team;W>\h thg 1o?:al data 
cbllectors during -the 'first ro.uhd of obslervatians attach' site: 

Data Confidentiali ty > ' ^ /' * ' ^'--^ - 

Confidentiality of educational 'data particularly >s^udent-related 
data -7 is a/subject whrch has, gqtteh a igrreat deal' of ^attention during the 
.past year. In order to addreSs ihis issue, 'v/e fee;! that , an onrsite coding 
system is the bft$t insurance against imprbper us^ of data. r 

At each svbe, the local data' co^llpctpr willj d/iring, the first week 
of the i975-77 scl^bol year, ^^sv'el.op a .keyeid; document which will assign a 
unique code' numbej/of each school, principal^, teather, aide, and student 
involved in the study. This key code contaiaihQ the unique' identifiers 
for all study participants v/ill be l<ept in a secured area in the local data 
collector/s office. Only o/ie. cl^iditional copy of the key code will be made. 
This copy. wiTT be seat in a dearly labeled sealed envelope to the Project 
Director.' The- Project Director will maintain the key codes from each site 
(unopened) in a secured area, as a -precaution a,gainst inadvertant destruc- 
. tion of'an on-site key code. Immediately upon termination of the project, 
both copies of alVkey codes wiTl be destroyed. 
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/^^) data 'collected, on-site will be coded with the unique identifier(s) 
of the study participants involved.. No data forwarded tq'the project core 
staff y/'i 11 contain name's of individuals* only codecf identifi-ers. 

' Administrative Procedufes> / ' ' \ 
In addition to the specif ic, data collection instruments identified 
in Section Ill/Tasks 1-4, other administ^ati ve reportijng procedures will ' 
be needed and will be the resppasibil'i ty of the local ^atai collector. 

There will be three administrative reports .wh-ich will^be maintained: 

♦ It 

-Time Allocation Log - . . , 

' t Expense Report ^ ' ' ^ ' ^* ^ , . 

• Daily ^AQtivity..Log ^ ' " . * ' , 

Jhe Time Allocation Log > , shown in Exhibit IV-l; .is a management con- 
trol mechanism by v/hich on-site project activity can be mpnitored. It.'also 
serves as a time she'et input into the implementation contr/^rtor' <; financial 
system for the local data collector. Each of the activi ties- .which the 
.local da*ta collector will , be expected to perform is enumerated (along with 
miscellaneous categories designed to include coding* and other "related 
activities). The Expense Report will be the s^aildard expense form for the 
implementation contractor, including the .nature and amount of each reim- 
bursable expenditure. The Daily Activity tog shown in-Exhi.bit IV-2, al- . 
lows the local data collector to maintain a record of his/her individua.1 
data col'lection contacts. This log will be maintained in coded form (i.e,, 
all schools and staff members will be identified only by t^eir coded iden- 
tifiers). Each of these administrative reports will be completed and sent 
to the local data collector's assigned Regional Coordinator (see Task 8)' 
on a bi-weekly basis. • . 
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Exhibit IV-2 



DAILY ACtlVSTY LOG 



name. 



date, 



district. 



1 TYPE OFACTIVITY: 



Interview 



••• LocBtion: 



(Schoo 1 j 



Testing , Observation 

(Staff) 



Comments: 



TYPE OF ACtlVITY: ' . Interview 
Location: 



~' (School) 



Testing Observation 
■ (Staff) ^ 



Comments : 



TYPE OF ACTIVITY 
Locat ion: 



I ntervfew 
(School) ' . ^ ' 



Test I ng. 



Observation 



(Staff) 



Comment's : 



TYPE OF ACTIVITY: 
► Locat i on: 



I ntervi eyi 



(School) 



Testing ,^ Observation 
(Staff) 



Comments : 



TYPE OF ACTIVITY 
Location: 



I ntervi ew 



(School) / 



Test ing Observat ion 

(Staff) " 



Comment sT ^ 



6 TYPE OF ACTIVJTY 
Location: 



Interview 



^ ^ (School) 



Testing Observation 
(Staff) 



Comments ; 



All School and^Staff enteries are to be denoted by QumericaJ identi'fiers 
only, , ^^Oi^ 
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Each of the adminis'trative instruments^/procedures describeci in t'his 
data colTection plan will be' d'iscussed' in somewhat greater detail in the 
Data .Collector's Manual described *ir> Task 8. . - 
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TffSK 7 - PROVID^E LOGISTICAL PLAN FOR PROGRAM finr.PFRflTTnrj 

— ^ t 

In this..task, we discuss several problems associated with obtaining 

« 

cooperation from study partitipants and outline a set of procedures and 
incentives designed to ensure this site cooperation. These perceptions, 
are based upon. the experience of project team members who have worked with ' 
•more than 200 LEAs during the last five years and upon related experiences 
in. several sinjilar large-scal-e Federally sponsored studies. In describing 
suggested procedures, frequent .reference will be made, lo' other^ "tasks . in 
whi'ch the procedures are descrit)ed in detail. 

General Impediments "^to Cooperation 

Many of the LEAs and publishers contacted during the design ph-ase have 
been somewhat concerned about site participation «in the proposed study.- 
Several LEAs reflectpd 

u-.- .^..^^o p,cY»uu:5 iiaciuMdi btUQieS; 

some expressed concern over the role of NIE in evaluating programs which 
are in lar^e part locally-developed and locally funded; still others/were.' 
somewhat 'concerned about outsiders coming into their dis-trict. Several 
, firms and publishers also expressed the following types of concerns: 

• ' Why is there a need to evaluate our program again? Inde- 

pendent evaluations have already proven its success'; 

• Will the contractor involveAsur consultants or Area/Regional 
manager, before going on-site, in order to "understand" the 
program thoroughly?; 

» 

e May we review instruments used to assess program success prior 
to final selection to ensure that the contr^tor is measuring 
what the program is designed to accomplish? 
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"Vihile the reasons for the concerns may or may not be valid, thes^e varying, 
perceptions must be taken into account. 

A second concern relates to the potential disruptive nature of .the . 
study, especially if a large amount of observation is to be conducted.. 
Tfie requirements to be placed upon the general administrative staff and 
principals (especially related to any data collection from parents) "ap- - 
pear to be of primary concern. ^ ^ ' " * 

A last general concern appears to stem from new legislatioar guide-- ' 
lines, and judicial decisions^ related to confidentiality of data. In any . 
nev/ area where uncertainty ofnnterpretation exists jmd individuals may 
be held personally liable, anxifeties will surface. Several LEAs said they 
would not participate unless it were made impos^ifele to^itdentify individual 
respondees; others, wanted their legal advisorsL^-to^-evtev^riy district data 
prior to public release^ Ithe implementa-Uw^Cvplitcactq^r must address these 
issues and demons tra^te stnct adherence to""contracto"r guidelines'* now 
being prepared by HEW. * ^ - ' . " 

* 3 

Specific Concerns 
'•Based upon our discussions with "LEA" staff members and consultants 
(including, a research-oriented Principal of a Title I school, a Federal 
, program director in middle-size city and program development specialists 
in an inner-city district) during the, site identification/initial selec- 
tion, a number of concerns related to either individualized or standardized 
programs were highlighted. 
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Individualized {"I") programs are more likely to be accustomed to 
outside visitors arid evaluation, especially if they have been^ selected 
for review by the USOE DRP or PIP programs. On the other hand, too 
great a study requirement on aji already over-burdened school staff could 
erode cooperation. Conversely, standardized ("S") program staffs might 
be very anxious and uncooperative, especially if they perceive their role 
as that of a control or comparison school. "I" program staff are likely 
to be more inquisitive (e.g., "How does our program compare to other sim- 
ilar programs in the study?") and to seek constructive criticism. Care 
must be taken to minimize the influence of outsiders and unintended ef- 
fects-generated during the period of observation. 

Since testing is generally more of an integral component of "I" pro- 
grams, concern over additional testing may be lower. The implementation 
contractor must be able to 'demonstrate a dear need to adrrj^inister the ad- 
ditional instruments., or the LEA staff may perceive the testing as being 
merely disruptive. '^St^ff in "S" programs are likely to be more suspicious 
of testing beyond minimal LEA requirements and more concerned about the 
improper use of test scores. To ensure LEA cooperation, the administrat^ion 
of any additional tests must be clearly justified to LEA staff; the use of 
resulting data must be limited to the study. 

* * * 

Procedures fbr Enlisting Cooperatipn . ^ , 

c Duri[i£^the design phase, several measures were taken to increase the 

probability of cooperation , and good working relationships with those LEAs 

finally selected for' participation. Data requests were limited to only 
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those neceasary to meet the purposes of initial site' selection. Although 

N time-consuming, a review of existing documentation of projects was used 

to complete the checklists. Thus, when telephone contacts were made, 

the questions asked were limited .to verification of existing data or a 

small number of direct inquiries, thereby reducing the length of converse- 
> 

tions. As an example, the final checklist completion for the three pro- 
grams irvone district took eight minutes, while review of documentation 
took pearly a half day. All LEA contacts were^open^, frank, and unofficial 
Several LEA staff -members wanted to get approval from their Board of Ed- 
ucation-prior to releasing any information,* We emphasized the tentative 
nature of their potential participation, indicating the steps 'which are 
being followed, including a formal information request from NIE or the 
implementation contractor next year. We suggest that the sampling varia- 
bles and general conditions for participation be determined as quickly ^s 
possible by NIE and that survey instruments similar to those in Appendix 5 
be cleared by 0MB and, sent to- potential sites as soon as possible. One 
person at NIE or the implementation contractor should be available by tele- 
phone ansv;er questions .regarding the data request and possible district 
participation. Since many of the candidate programs ''are at sites partici- 
pating in other federal efforts, this coordination at the Federal level wi 
also tend to minimize confusion. 

The primary means of enlisti)hg flte^support and cooperation of LEA 
staff is through careful selectioin of local dat^ol lectors who will repre- 

':he district and the project. The 



sent the primary liaison between 
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specific procedures and qualifications of locaT data collectors who will 
represent the principal liaison between the district and the project." The 
specific procedures and qualifications of these field staff will be dis- 
cussed in Task 8. It will suffice in this task to indicate that among the 
-qualifications for^the local data collectors will be the abili'ty to work 
closely and cordially with .LEA staff at the district, school, and class- 
room leveT. It is strongly felt that the usCU local data collectors, 
familiar with the policies and traditions of'tKe district as well as, in 

many cases, its personnel, is a major step in^avoiding the "outside eval- 

■ » 

uator" stigma frequently associated with such field studies. 

-At least two types of incentives for copperation may be provided to 
LEA personnel. The first type o-f incentive is informational in nature: 
the provision of informational feedback to the LEA on the performance of ' 
their programs. Two major concerns must be exercised when considering ^ 
such an incentive. The first is thafNIE is the' primary consumer of the 
study's results and, that any ancillary use of study data or results must 
meiet with NIE's full approval. ■ The second concern is that any early feed- 
|)ack of data or results may have a confounding effect on the study by in- 
fluencing the subsequent behavior of study participants. It is our recom- 
mendation that, with NIE approval, project results iV draft form be pro- 
vided to the participating LEA staff as soon as possible after the post- 
test is administ6!red. 

The second type of Incentive is monetary in nature. It would pro- 
vide a nominal sum to participants in gratitude for their cooperation. A 
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number of alternative means of providing this incentive present themselves 
One technique, frequently used, is simply paying teachers a' small sum (per 
haps $10) for submitting to an interview. Although this does distribute 
funds to teachers, it is unlikely that, on an individual basis, this small 
sum would provide any great incentive for proj'ect cooperation. Another 
means of providing incentives for staff cooperation, and one which we 
recommend, is to provide the incentive on a program-wide basis, distri- 
buting a total of perhaps $250 per program to be allocated equally among ' 
the participating school buildings. This would provide the funds as an 
addition to the regular resources available to the students participating 
in this. Care must be exercised, when distributing this jjioney, not to dis 
rupt the LEA'S normaWinancial system. -Close coordination with the super 
intendent's district- lelcej designee-will be observed. A second concern 
is t^at the introduction of such dollar incentives might influence- the 
conduct of instruction by introducing additional resources into the class- 
room. Although this is unlikely, the possibility may be. addressed by not 
distributing the incentive funds until after the study is complete. 
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TASK 8 -- DEVELOP STAFFING PLAN -^-^^ 

In this task, we specify the staffing plan which will be required to 
conduct the implementation of the study bf individualized Instruction in 
the manner described in this report. Ourpiscussion "of this task can be 
broken down into five principal -subareas^ ~ ' - 

• Project Organization . ' 
t ' Project Person Loading 

• Selection of Field Staff 

• Training of Field Staff - \' 

• Data Collector's Manual .■ . ^ 

As indicated in our discussion of Task -5 , 'the last. three of these subareas 
might also be considered part of the data collection plan. 

Project Orqantzation . - ' • ^ ' . 

Exhibit IV-.3 depicts a suggested orqafiization chart, for the «;iiccess- 
ful implementation of the study. This activity-oriented structure reflects 

« 

the three major tasks to be performed in the study ' s implementation pha-se 
(i.e., Sampling, Data Collection, and Analysis) as second level management: 
The nature of the position and the necessary/qual if ications bf each 
■ member of the core staff will be discussed in the following-paragraphs: 

Executive Supervision 

As with any large-scale project, it is desirable to have top level 
• oversight from someone at a high level within the management of the con- \ 
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EXHIBI7 IV-3 



ORGANIZATION STRUCTURE 



., NIE 
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MANAGER 



FIELD TEST 
DIRECTOf^ 



TRAINING 
DIRECTOR 



PROJECT 
DIRECTOR 



DATA 
COLLECTIOfI 
iIAfiAGER 



REGION I • 
COORDINATOR 



SEN'IOR 
TECHNICAL 
ADVISORS 



^ 5 
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MANAGER 



REGION II 
COORDINATOR 



REGION III 
COORDINATOR 
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tractor.^ The role of the person providing this Executive Supervision would 
be: 

• to make official commitments on the part of the implementa- 
tion contractor, 

• to serve as liaison between the contractor and the NIE Con- 
tract Officer in contractual matters, • "* " ' 

• to commit the apjDropriate staff of the implementation zGfT- 
tractor to the study, and , ^ ■ 

• to serve in ah advisory capacity on technical matters to the 
Project Director. . 

Executive Supervision of the study .would nave little or^^no^ technical 
involvement and would participate at a. relatively low level of effort (perhaps 
three days per month). ' The person filling the Executive Supervision "role 
would have such a title as Project Administrator, Partner-in-Charge, Vice- 
President, or even President within the implementation contractor's manage- 
ment organization. 

Project Director 

The Project Director will be the person directly responsible foT^ 
technical performance of the study. The principal functions of the Project 
Director will be: 



to s-erve as the major point of contact between the NIE -Proj- 
ect Officer and the implementation contractor's project team, 

to allocate the personnel and material resources of the proj- 
ect team.to best serve th^ successful implementation of the 
study, 

to jjrovide technical leadership in the design, development, 
and conduct of project tasks, 

to monitor the day-to-day progress of the study, and 
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• • to serve as principal investigator/author of the final eval- 
uation report. 

• » 

The Project Direator sfiould be experienced in both compe^nsatory edu- 
cation and, the evaluation' education programs. Even more important, he/she 
should be experief^ced in the management. of simiU*- ^large-scale evaluation 
studies. In the interests of project quality and credibility in the aca- 
demic community, it' would be desirable if the Project Director held a doc- 
toral degree or ^ive years experience in directijig similar large-scale tech- 
nical studies. The Project Director should devote at least 75% of his/her 
time to the project for the duratix)n of the study. 

\i ■ 

Senior Technical Advisory Panel 

In a project such as this study of individualized instruction, it is 
extremel^y desirable -for the Project Director to have at his disposal a number 
of well-known experts in the var%us disciplines associated with the study, 
as well as others whos6 interests are closely assotiated with the study. 
These advisors would assist the project team by* providing special expertise 
when needed and general project review and criticism. 

This advisory panel would have spe^"ial expertise in such areas as in- 
dividualized learning, compensary education, tests, measurement, and eval- 
uation.. Its members could also include persons chosen for their insight 
into the political as well as technical aspects of the project. A suggested 
size for this panel would be approximately ten people who would devote their 
efforts to specific tasks designated by the Project Dirpct.nr. 
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Sampling Manager ' ' , V ^ 

The first task in the impTementation phase of-; the study, (see' Task .9) ' 
is the selection of the final sample from a'mong-^a nominated group based 
upon existing program documentation and sampling data gathered during the ' 
early stages of the implementation phase. The SampTing Man,ager should have' 
experience in sampling theory and techniques, .as w?1l as in' design and anal- 
ysis of projects involving LEA's. The Sampling Manager should hold at least 
a master's degree in education, evaluation, quantitive methods, or some 
related discipline. The Sampling Manager will have heavy involvement during 
the early stages of the project with very little work after the sites have 
been chosen-. The nature of the functions of the Sampling Manager and the 

> 

Analysis Manager make it possible for one person to*fill both roles. 
Data Collection Manager 

The Data Collection Manager will have responsibility for the conduct 
of the second and third tasks (se'e Task 9) of the implemen'tation phase of 
•the study, including the selection and training of .field staff and the^man- 
agement of the overall data collection effort. The Data Collection Manager 
should be experienced in survey research and data collection procedures and 
in the logistical aspects of interacting with LEAs. The Data Collection ' . 
Mdnager should hold at least a master's degree in education, survey research, 
or some related field. The' Data Collection Manager will essentially devote 
full time to the project for its duration. 
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Analysis Manager 

The Analysis Manager will be responsible .for the fourth task of the 
implementation phase, data analysis. The Analysis Manager should be ex- 
perienced in the evaluation 'of education programs in general and in the 
testing and measurement of educational achievement and attitudes in partic- 
ular. The Analysis Manager should hold at least a master's degree in one 
of tha^.e disciplines. The involvement of the Analysis Manager will be 
fairly uniform during the course of the project, with particularly heavy 
involvement during the finalization of data collection instruments and the 
reduction and analysis of actual data. As indicated, the Analysis Manager 
might, if qualified, also fill the role of Sampling Manager. 

Field Test Director 

The field test of data collection instrumenirs is to be'conducted 
during the Spring of 1975. One member of the project team, with experience 
in the various data collection aspects of public education, should be desig- 
nated as director of that field test. He/she would then have responsibility 
for-all actiV^fties involved in the fie}d test and for ensuri-ng that field 
test results .are, fed back to the apprdpniate members of the project team, 
as well as coordinating the field test resu.lts with th^ 'OMB documentation 
package. The Field Test Director will hav-e heavy project. involvement during 
the first four months of the implGmentati on- project. The position becomes 
inoperative when the field test is ^completed and .the results reported. The , 
Field Test Director may also be qualified to fill the role of one of the 
Regional , Coordinators . ■ i ' \ - " 
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Training Director • ' 

A crucial facet of the data collection effo^ is. the trailing of 
field staff. Local data collectors will be trained in two'formal training 
.sessions which must be properly planned and executed. 'One member of the 
project team with 'expertise in .this area should be specified as Training 
Director. -The training'Dir'ector will devote most of his effort duri.ng the 
Summer of 1976. ' ' • V . ' 

Regional Coordinators . , " 

. , In o.Mer to eff^tively monitor the large magnitude of data which is' 
expectecl to flow frotp the sites and to ensure that the high'est possible dV 
gree of data reliability is achieved in the data collection efforts, three 
Regional Coordinators should be assigned from..the implementation contractor's 

staff. These Regional Coordinator? win work ciojely with the Iccfil 
■data ^collectors in their regions. They may also team with the local- data col 
lectors for the actual collection of some data. Each Regional Coordinator 
will, have responsibility for monitorihg and assisting approximately eight 
•sites. The exact assignment of sites and programs will, of course, await 
find.1 site selection.. These Regional Coordinators will devote essentially 
full time to the project during the data collection period (i .e. , the 1976-77' 
school y^ar). ' • . ' . * 

. Project Person Loidiiiq - ' - . ' ' • ■ 

Based upon our assessment o-f the project requirements, the suggested :' 
scope of the a>ialy$is, and the '^stimerted size of thei sample, we have made 

V 
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'a task-by-task estimate of the level of effort required' by each project 
team member. Exhibit IV-4, following this page, ref lects'-the^e person- 
loading estimates. 

Selection of Field Staff ' \ 
A critical component of the overall project staffing is theselec- • 
.tion of properly experienced and qualified field staff as local data col- 
lectors. As ^discussed in Task 7, we believe the use of professionals with 
'community ties and acquaintances in the scliool districts chosen as' study 
Sites wi;i greatly enhance the level of cooperation received from, site per-'v 
sonnel. ^'it is Important, however, that the'se field staff be more than merely 
acquainted with the schools and its personr>€l . They' should have familiarity 
with compensatory education programs and should aware of the political 

....f. . . t>« w . v^..^ I • Luui,u I, I UM evaluations bucii db cm s One. 
Since ft is not possible at this time to specify with certainty the 
number of study^units (classrooms) includpd at eabh s'ite w$ h39£ mde esti- 
mates of time requirements based upon approximately 40 classrooms per site 
Based upon the data collection- activities noted in Exhibit IV-4, this esti- 
mate.ineTudes the following field staff requirements during 1976-77 school 
year- for. an ^average site. " - . . 
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ESTIMATED PROJECT PERSON LOADING 



Exhibit 



person days 



tasks 



TASK 1: SELECT FINAL SAMPLE 

1.1 Administer Questionnaire 
1 ..2 Conduct "Verification Visits 
1 . 3 Select Sample . 

TASK 2: SELECT AND TRAIN DATA 

COLLECTORS 
. 2.1 iRterview'ilooal Candidates 
2:2 Sele(5t •^)ata* Collectors 

2.3 Develop Trainlftg Materials 

2.4 Conduct Training Session- 1 

2.5 Conduct Training-Session II 

TASK 3: CONDUCT DATA COLLECTION 

3.1 Finalize Instruments 

3.2 Prepare for Data Collection 
• 3.3 Conduct Interviews 

3.4 Conduct Testing - 

3.5 Conduct Observations 

TASK 4: CONDUCT DATA ANALYSIS 

4.1 Reduce Data 

4.2 Obtain Scored Test Results 

4.3 Clieck Data for Consistency . 

4.4 Concjuct Specific Analyses 

TASK 5: AePARE FINAL REPORT 

5.1 Prepare Draft Final Report 

5.2 Prepare Final Report 

total 



percent of time^ 











0 
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PROJECT 
DIRECTOR 
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25 


5 " 


5 


40 




1 


\ f 


9 


20 . 


5 




40 ' 




4 


20 


16 


20 


5 




10 








4 


— 


5 


5- 


40 





2 




4 
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60 
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35 


10 
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10 


180 


500 
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70 




70 


10 
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60 
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5 


5 
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10 
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180 




6 




10 
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10 


30 
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10 


20 
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20 
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95 
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1 14% 




86% 
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total project person days 



7200 
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* Based on total project duration of 360 work days 

** If a' qualified person is available^ the role^ of Sampling Manager and Analysis 

Manager may be filled by the same person. 
*** Including Regional Coordinators oor* 
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Interviewing 

Interviewing-Misc. 

Testing 

Testfng-Misc, 

Observation 

Observation-Misc. 

Training 

Administrative 



Person-Days 
40 
15 
10 
^10 
MO 
10 . 

10 : 

35 

170 TOTAL 



These estimates represent average time requirements for the local data t-ol- 
lecton at 'each site tcr conduit eaCh'of the three primary data .collection ac- . 
tivities (i.tli Interviewing^, Testing, and Observation), plus miscellaneous 
tima for each activity to aVhJw for coding, consistency checking, 'and other • 
similar^duties related .to the specific activities- Additional provisions ' 
are made for time to attend the two training conferences, as well as time 
allocated >to general administrative and coordination activities. 

Using these estimates as a guide it appears reasonable, to assume that 
one person, almost full time, during the 1975-77 school year will suffice for 
each site. Modifications in this estimate must, of course, be made at the 
time the precise sample pf programs and sites is finally selected. In light 
of the general oversupply of teache^in the United States, it is highly 
likely that qualified local data collectors can be identified and enlisted 
for approximately >a regular teacher's contract "period and salary. 

A number of channels exist for the recruitment ol" qualified field staff. 
The primary source, of course, would be the personnel office of the local 
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school district. In general,, these personnel offices have far more appli- 
cations than^available openings. Districts may also be willing to give a 
year's leave of absence to a particularly qualified staff member wishing 
to participate in the study- Another possible source of field staff would ' 
be the education departments of local colleges and universities, which could 
identify current graduates and other local people active in the educational 
community. A third source of information on potential field staff candidates 
would be professional contacts of jnembe^s of the implementation contractor's 
proj^ect*team. ^ ^ » 

FYocedural ly, candidates for field staff positions should be identified 
as soon cis possible even, before final site selection is made. Resumes of 
potentia^l .ca^ndidates should be solicited from all sources as the final sample 
selection process is continuing. The verification site visits to be made 
prior to final selection represent a good opportunity for the implementation 
contractor to interview-'tandidates . Immediately upon completion of the selec- 
tion. of sites, (by July 1976) commitments' should be made and contracts 
, .negotiated. * * 

Traihi'ng of Field Staff 

In order for the study to have an effective data collection arm, the 
training of field data collecto^ is crucial. A great deal of time will be 
spent by the project team (the Training Director, jn particular) in develop- 
ing' a high-quality presentation of two training devices, agendas for the-'^wo*" 
training sessions and a Data Collector's Manual. These two training devices 
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will be parallel in nature,* the first, serving as initial lesson the 
second as a continuing guide throughout the' project. - 

Each ftf these two training tools will be divided into four basic 
subsections: . 

• General information in which the purpose of the study is ^ 
outlined and the role of the local data collector is specified, 

Protocol --'in which political considerations are described 
* and^apprDpriate procedures detailed,^ 

> 

t*^ Administrative — in which the basic reporting, record- 
keeping, ^confidential ity, and communication aspects of the 
project are descriJifid, with particular emphasis on the 
> written requirement^ and administrative logs (see Task 5), 

• Data collection — in which each of the specific data col- 
lection 'instruments is described and the procedures for 'its 
use detailed. 

Appendix 14 of this report is a draft of a suggested'iData Collector's 
Manual. In Appendix 14, some sn^cific nrocedu^^es are d«^<" 
the particular instruments are rrot completely defined; general procedures 
are. outlined. In addition to the information contained in the Data Collec- 
tor's Manual, the training sessions will contain provisi'bn for a great deal 
of practice infusing each type of instrument. This practice, under simulated 
classroom and interview situations, is essential if inter-rater reliability 
is to be high. 

4 

In order to meet the tight time constra,ints imposed by the project 
schedule and to ensure, the highest possible degree of data reliability, the 
.training of local data collectors will be conducted in two five-day training 
.sessions during August 1975. The first session will accentuate general, pro- 
tocol, and administrative aspects, as well as interview and testing instru- 
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ments and procedures; the second will concentrate heavily on classroom 
observation procedures,. The locations for the two training sessions will 
be determined once the specific study s.ites are selected. They will ^e 
chosen for their central location or for other economic factors. ' , 

The training rec^eived by local data collectors will be reinforced by 
the Data Collector's Manual and by close interaction between the local data 
collectors and their designated Regional Coordinator. As indicated in our 
discussion of Task 6, the Regional Coordinators will assist the local data 
collectors by teaming with them on some of their initial interviews *and 
observations. ^ * • ■ 



TASK 9 - DEVELOP IMPLEMENTATION SCHEDULE 



In this task, we identify the specific tasks associated with the im- 
plementation phase of the individualized instruction study and describe the 
proposed schedule by which the study may be compTeted before t\)e Congres- 
sional deadline of July 1977. 

♦ 

Task Descriptions ^/ 

Task 1: Select Final Sample — This task encompasses tnose subtasks 

which relate to the selection of the districts, schools, and classrooms to 

be included in the study. 

1.1 Administer Sampling Questionnaires Assuming 
that the sampling questionnaires for district 
and school -level personnel (see Appendix 5) 

will be submitted to 0MB for clearance prior ^ 
to implementation contract award, they will 
be sent to the ^appropriate LEA personnel in 
accordance vnth the procedures described in 
/ Task 2. ' 

1.2 Conduct Verification Visits — For those sites 
at which additional or verified information is 
required. On-site visits will be made to com- 
plete and verify the data requirements for all 
candidate programs. 

1.3 Select Sample Based upon all available infor- 
mation , including existing program documentation. 
District Survey I data, sampling questionnaire 
responses, and sVte visit* data, the final sample 
of study partioljpants will be selected as indi- 

v,^ ' cated in the Task 2 discussion. ^ 

Ta.sk 2: Select and Train Data Collectors'-- This task includes all 
activities associated with the recruitment, selection, and training of the 
field staff which will perform the on-site data collection duties. 
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2.1 Interview Local Candidates — During verification 
visits (Subtask 1.3), potential candidates for 
position as local data collector will b%"^'sol ici tecl 
and interviewed. 

2.2 Select Data Collectors — When the final selection 
of sites has occurred (Subtask 1.4), the local data 
collectors for each selected site will be chosen. 

2.3 Develop Training Materials — The materials and 
techniques to be used in the training of local 
data collectors will be developed. 

2.4 Conduct Training Session I The first training 
session, including a general overview of the 
project, interview instruments and techniques, and 
testing procedures, will be conducted. 

2.5 Conduct Training Session IL-- The second training 
session, on classroom observation instruments and 
procedures will be conducted. 



• Task 3: Conduct Data Collection This task reflects the three 
basic data collection activities the study and the corresponding pr 
inary activities, 

3.1 Finaltze Data Collection Instruments The 
specific instruments that wil'l be used to col- 
lect data' (including interview, testing, and' 
classroom observation instruments) will be 
finalized and, along with their corresponding 
documentation packages, submi tted^to 0MB for 
clearance. Thi^ 'subtask also in^ludfes^the 
field test of the instruments. ^\ 

3.2 Prepare for Data Collection Contacts and ^\ 
schedules for the conduct of data collection ^ 
activities' are made. 

3.3 ^ Conduct Interviews Interviews with LEA will 
. ^ be conducted at the beginning of the school 

- year and again toward the end of the school 
year. 
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3".4 Conduct Testing -- During the early autumn of 

1976, student pre-tests in both achievement 4 
and affective areas will be administered. Cor- 
responding post-tests will be adm^ffistered dur- 
.ing April 1977. In both pre and post-tests,' 
particular care will be exercised to ensure ef- 

• fective coordination of such administrative mat- 
ters as ensuring the appropriate space and 
developing a quick response coding plan. 

3.5 Conduct Observations -- Classroom observations 
will be conducted throughout the period bet- 
ween September 1976 and April 1977'. 

Task 4: Conduct Data Analysis -- This task includes the reduction, 
preparation, and analysis of data. It should be noted that, in order to 
facilitate the timely analysis of data, data reduction, verification, and ^ 
analysis, will be conducted on a continuing basis during the 1976-77 .school 
year. 

4.1 Reduce Data -- The raw data will be converted 
into a fprmat suitable for analysis. 

4.2 Obtain Scored Test Results — Pre- and post- 
test scores will be obtained from the scorers. 

4.3 Check Data for Consistency — The reduced data 
must be ^checked for internal .consistency prior 
to analysis. 

4.4 Conduct Specific Ana'lyses — The analysis ac- 

• ' tivities outlined in the analysis plan (Section 

Ill/Task 5), will be' conducted. , 

Task 5: Prepare Final Report - This task reflects the preparation 
of a final report to NIE. 

5.1 _ Prepare Draft Final Report — The final report 

will be prepared in draft form: ' 

5.2 Prepare Final Report — Based upon NIE recommen- 
dations, the dr^ft final report will be revised 
into the final version. 
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Project Schedule 

Exhibit IV;5 displays, in the form of a GANTT chart, the schedule 
by which the project tasks will be conducted. Exhibit IV-5 expands the , 
schedule into the form of a PERT network. The final chart. Exhibit IV-7, 
describes the specific products and reports to be submitted during the 
implementation phase of the study. 
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LIST OF DELIVERABLE PRODUCTS 



Item 



Date 



1. 



Final Data Collection Instruments 



1 June 1976 



• Interviev/ Instruments 

• Observation Instruments 

• Achievement Tests 

• Affective Instruments 

0 Complete 0MB Documentation 

2. Training Materials 1 August 1976 ; 

6 Training Conference Agendae 

• Data Collector's Manual 

4 ' 

3. Interim Report . 10 /lovember 19T»^^ 

t i^roject Status Report 
8 Interview Summary 
ft Tp<; ting Si^mna ry 



Monthly Progress Reports, in letter form, will be submitted throughout 




20 June 1977 



t Project Summary 
t Analysis Findings 
0 Recommendations 



final Report 



31 July^ 1977 



the project period. 
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